TLDR: Since November 2021, Google has enforced new storage access limitations for apps published on its Play store which prohibits direct storage access on Android 11 and above forcing QField to adapt and rely on importing projects and datasets to access those.
If you are a QField beta user on Android 11 and above, you might have noticed a significant change in the way the app is handling storage in the latest set of betas released in early February of 2022. This blog post will go over the changes, explain why those had to be made (looking at you, Google), how to work in this new paradigm, and showcase some new benefits from the hard work done by OPENGIS.ch’s geoninjas.
It’s all gone! How can I access my projects and datasets?!
Starting with Android 11, apps are denied full access to main and external storage content. For QField, this means direct access to projects and datasets transferred and/or downloaded into storage folders is not possible anymore.
To work within this new confine, QField now has to import project folders or individual datasets into an app-dedicated storage location where Android allows for unrestricted read/write access.
Practically, this means that instead of being shown and having access to the full storage tree when clicking on the “Open local files” button, users are now shown a set of new folders named ‘QField files directory’, ‘Imported datasets’, and ‘Imported projects’ as well as a drop-down menu accessible via a top-right three-dot button.
Import project from folder
When importing a project from a folder, users will be asked to grant permission for QField to read the content of a given folder on the device’s storage via a system folder picker. When the folder is selected, QField copies the folder content (including its sub-folders) into the app’s ‘Imported projects’ location. Users can then open the project from there.
Re-importing a given folder through the drop-down menu action will overwrite preexisting projects given an identical folder name. That allows users to be able to update projects.
Note that feature editing, addition, and deletion will be saved into the imported project’s datasets, not in the original folder selected during the import process. More on how to find and handle those project datasets will come later in this post.
Import project from ZIP archive
Having to adapt to Google’s new set of rules did not come without its benefits. Users can now easily transfer projects into a given device by compressing the project content into a ZIP archive and having QField import that compressed project automatically. This can greatly ease remote deployment of projects by being able to send a single file to users.
Import dataset(s)
QField can also import individual dataset(s). Users will be asked to select one or more files via a system file picker, which will be copied into the ‘Imported datasets’ folder. Users will have to ensure that all sidecar files are selected when importing (e.g. a shapefile dataset would require users to select the .shp, .shx, .dbf, .prj, and .cpg files).
Just like imported projects, editing of datasets will be saved into the imported datasets, and not reflected in the original files.
Alright, but how can I retrieve modified projects and datasets?
Imported projects and datasets can be accessed directly using a USB cable. The location on storage is displayed in the top navigation bar when opening a local file.
On most devices plugged into a computer via USB cable connection, the path will be <drive>:/Android/data/ch.opengis.qfield/files/ where you will find both the Imported Datasets and Imported Projects folders within which your edited content will be located.
However, we’ve also added a nice new ‘Send to…’ functionality that allows for users to share and send datasets straight from QField using Android APIs. This allows for the sending of edited datasets directly to third party apps (Gmail, Drive, Dropbox, Nextcloud, your favourite messenger app, etc.).
Is direct copying via USB cable gone altogether?
Users can still avoid going through the import process by copying files via a USB cable connection directly into the QField app’s files directory. As mentioned above, the location on most devices will be <drive>:/Android/data/ch.opengis.qfield/files/.
What are the benefits from these changes?
Working out a functional solution to meet Google’s newly-enforced restrictions did not come without its benefits.
On top of what was already covered above – importing of compressed project ZIP files and sharing functionalities – QField is now fully integrated with Android’s cross-application document sharing APIs. This means that users can now directly open projects and files sent to them via their favourite browser/email/cloud/messenger app without the need to first download those files onto the device.
Altogether, the newly-coded importing mechanisms and integration with Android document APIs don’t only improve the ease of use for the average person, it also makes viewing and editing spatial datasets on QField far more secure. The imported projects and datasets reside in a location with access limited to QField only, meaning that its content is inherently far more protected from malicious access from third-party apps.
Why were these drastic changes needed?
As mentioned in the introduction, the changes were needed to comply with a set of new Google Play policies that came into force in November 2021. Users can read more on Google’s rationale on this page https://developer.android.com/google/play/requirements/target-sdk.
As part of the enforcement of these new policies, Google came up with an arbitrary mechanism to whitelist some apps which allows those to retain full storage access given the user explicitly allowed for it. We here at OPENGIS.ch believes QField had ample justifications to be whitelisted, however, Google’s appeal process judged otherwise after a series of email exchanges detailing our reasoning. While we have so far lost this argument with Google, we will continue fighting for our users and for their freedom to choose. If by any chance you have a good contact at Google that might be willing to listen to our reasoning, we would be grateful if you’d get in touch with us.
We hope this clarifies the recent changes and helps QField users adapt to those.
24 Comments
Giovanni Manghi · 2022-03-05 at 11:17
Hi Marco, thanks for this post. Without an Android 11 device at hand right now, I’m trying to understand the implications in a scenario that involves QField and synchronization of project files and datasources (gpkg, shp, etc) between an Android device and a desktop/laptop (no qfieldsync involved). Does this mean that if a (fpr example) a GPKG datasource will change on the mobile device there is no more automatic synchronization to the laptop/desktop (and vice versa)? What about the NC mobile app, would not be possible to manually add to it the path :/Android/data/ch.opengis.qfield/files/ so to still allow automatic sync?
Mathieu · 2022-03-06 at 05:41
Giovanni, with regards to nextcloud mobile app, on Android >= 11, the system doesn’t allow 3rd party access to an app’s files folder, so that won’t work unfortunately. Simply stated, on Android >= 11, there is simply no way to do project folder synchronization. That’s one of the main reasons why we think QField should be granted all files access whitelist on Google Play store.
You can still do direct USB cable transfers to/from the storage path (/Android/data/ch.opengis.qfield/files/).
In addition, you can send individual dataset files to a drive/nextcloud/dropbox folder via the “send to…” action action (described above). So say you have a project that’s hosted on nextcloud, you can use the “import project folder” action to copy the content into your QField project, then regularly use the “send to…” action to send an updated geopackage dataset back into the nextcloud project folder.
Mathieu · 2022-04-13 at 07:55
Giovanni, additional functionalities have been implemented in QField 2.1:
1/ QField now allows for exporting a whole project to a user-picked folder on your device or directly onto a NextCloud account. This should make it infinitely easier to upload updated project content.
2/ Users are now able to compress (into a .zip container) and send project folders directly from QField and send that file using Android’s sharing functionality. You can in one click zip up your data folder or your photos and send via email, messenger app, etc.
Giovanni Manghi · 2022-03-06 at 09:52
Mathieu, hi, thanks for the reply. I got my hands on an Android 11 device and checked myself that indeed if things stay as they are, any automatic synchronization is gone. What a shame for Google. A couple of notes.1) As QField (as any other app) can access only its own scoped storage, it seems to me that it could miss a functionality to delete files from this storage. I tested the following a) imported a project from folder > Qfield imported everything from that folder b) from the original folder I deleted a file c) in Qfield I re-imported again, and the file that was deleted was not deleted from QField scoped storage. 2) you could check if esri’s mobile thing had the exemption from this new policy, if they did it could be a strong argument in favor of QField getting it too. Cheers!
Peter · 2022-03-07 at 10:12
Thank you, guys, for your great work during the last months and this eye-opening post! I’m very much looking forward to QField 2 and all its great new functions ? Sad to hear, that Google seems to be unwillingly to give you the permissions you need.
For us, it will be quite a big change in our way to import and export packages to QField. Unfortunately, it will not be possible to connect via USB in our company. So, we will have to rely on the new import and send-to functions. While importing projects from folders seems to be not a big deal, I’m wondering if it is possible to export folders or zips via the “Send to…” functionality. We often have projects with plenty of photos taken, which have to be shared or exported as well.
Thanks and keep up the great work!
Boswachter Marc ? (@Boswachter_Marc) · 2022-03-10 at 23:22
Does this also affect QFieldCloud projects with the QField beta version v2.0.12 and the latest 0.13 release? I am using Android 12 and have sync problems on tablet and phone. Eventually, I will make an issue on github. Thanks for all.
Marco Bernasocchi · 2022-04-04 at 11:38
Hi Marc, no if you use QFieldCloud you’re good since all is written in the QField private folder
jfmoyen · 2022-04-04 at 10:06
Dear QField Ninjas,
Thanks for the explanation. I’m in the process of moving to QField (from FieldMove), and trying to set up things in a way that works for me. Similar to many, this permission business is annoying – I’m using a sync app (SyncThing) that, being a regular app, has no access to QField’s directory ( /Android/data/ch.opengis.qfield/files/ ). This is quite annoying, as it means I have to rely on USB transfer (or on moving to Cloud, which I may also explore, but that’s a topic for another day).
However there is something I fail to understand. On the same device, FieldMove stores its data in /FieldMove, and this is a location I can both read and write, from FieldMove as well as SyncThing (and I’m indeed sync-ing this folder with other devices). I also see a /qfield folder (it contains empty auth, basemaps, fonts and proj sub-folders): wouldn’t it be possible to use it to store the data ? What am I missing ?
Thanks !
Mathieu · 2022-04-04 at 15:30
Greetings,
Regarding FieldMove, the reason why it can still store/access data in /FieldMove is because the app hasn’t released any update since February of 2021. The new storage policy applies for new app and updates to pre-existing apps starting November of 2021. This means that for us to provide new features and important fixes to QField, we have to abide by those new storage restriction. FieldMove will find itself in the same situation whenever they push a new update to their apps.
The /QField folder was created using an older version of QField (up to 1.10), which were published prior to the November 2021 deadline. If you remove that folder and launch QField >= 2.0, you’ll notice the folder isn’t re-created then.
As for synchronization with SyncThing (or nextcloud, or {insert your favorite sync app}), we feel your pain. In QField 2.0, if you want to synchronize individual datasets within your project (say a geopackage), you can use the ‘send to…’ action on the desired datasets and select SyncThing.
In QField 2.1, we will add a function to compress a whole project folder into a ZIP and export that to a folder / SyncThing, etc. That will hopefully make your experience better there.
jfmoyen · 2022-04-04 at 16:23
Thanks Mathieu.
I was not aware of the subtleties with pre-existing apps and dates. Yes FieldMove does not update often, which is one of the reason I’m trying to transition.
The “send to..” {sync app} may be a decent workaround, possibly an even better solution in fact, because it avoids syn-ing the temporary files created around a gpkg that is being edited (-shm and -wal) and gives me control of what I “push” to my reference copy. However my projects are photo-heavy (> 350 photos in the current one, typically 20-30 shots per field day), so I also need to export the DCIM folder as well. Sending photos one by one would be a hassle, and really error-prone.
Indeed, the possibility to zip and send a folder would help. Still, 350 photos is more than 1.2 Go, which is big enough to try and avoid duplicating/processing/moving things to often ! The solution would be the possibility to “send to” only the photos that are not yet in the target folder, if you see what I mean – i.e. an an option “merge with target folder” or something similar.
Anyway, thanks for the answer…
Mathieu · 2022-04-04 at 18:10
No problem, and I hope you’ll find a new home in QField. This storage situation is not ideal but I’m sure that we can find adequate workflows for all users. Thanks for you’re feedback and happy field mapping!
Dubhain · 2022-04-07 at 08:14
Would publishing on F-Droid make any difference? I don’t know if it’s just coincidence, but all the apps I have installed from F-Droid seem to have no trouble reading and writing from shared storage. Plenty have updated since February. I can’t believe they’ve all been granted special permission fro Google, but your hasn’t, so I though maybe their being on F-Droid made a difference?
Mathieu · 2022-04-13 at 07:56
jfmoyen, additional functionalities have been implemented in QField 2.1:
1/ QField now allows for exporting a whole project to a user-picked folder on your device or directly onto a NextCloud account. This should make it infinitely easier to upload updated project content.
2/ Users are now able to compress (into a .zip container) and send project folders directly from QField and send that file using Android’s sharing functionality. You can in one click zip up your data folder or your photos and send via email, messenger app, etc.
Attila Ferenc · 2022-04-12 at 11:38
We use FolderSync to sync files and folders with a NextCloud server in a Hungarian national park. You can add permissions to r/w the /Android/data folder. You can use it with dozens of cloud services (Google Drive, OneDrive, WebDav, etc.), and finely adjustable.
Giovanni Manghi · 2022-05-04 at 16:09
@Attila Ferenc, could you expand about you use this FolderSync app and how this can help mitigating the new folder permission problem in QField?
Giovanni Manghi · 2022-05-04 at 16:11
If I’m not wrong it has been attempted to contact Google trough an employee and see if they could raise an exception for QField, how did it went? Got any feedback?
Giovanni Manghi · 2022-05-04 at 16:14
Mathieu. you say that in QField 2.1 will be possible to “exporting a whole project to a user-picked folder on your device or directly onto a NextCloud account. This should make it infinitely easier to upload updated project content”, and this is great, but what about the other way around?
Mathieu · 2022-05-23 at 06:36
Giovanni, the other way around (i.e. importing a nextcloud project folder into QField) as been available since 2.0.
VxTedxV · 2023-06-06 at 07:12
Votre site web ne permet pas de voir les commentaires des autres utilisateurs !
Roland · 2024-07-18 at 12:32
Hi, I am late to the party. I run a data collection operation using tablets in remote areas of the countryside. We have 8 tablets running Android 10 and have become used to being able to collect data in the field and sync it back to the office. I sync Imported Projects to a Dropbox.
Recently, things were going so well, we decided to invest in more tablets for the field data collectors. The new tablets run Android 13 and I can no longer access my own data to run the sync operations.
Big frustration.
Is there scope to allow QField to sync data to a personal folder in Documents?
Can the existing export utility be amended to export whole projects including pictures and attachment files?
Is QField Cloud available to run from my own server without using cloud storage?
Mobil QGIS förändras – Geosupportsystem · 2022-03-09 at 08:49
[…] Mer om förändringarna här: https://www.opengis.ch […]
QField Storage Access on Android – GeoNe.ws · 2022-03-22 at 15:25
[…] https://www.opengis.ch/2022/03/05/qfield-users-sit-down-we-need-to-talk-about-storage-access-on-andr… […]
QField 2.0 Arctic Fox is here – OPENGIS.ch · 2022-04-05 at 07:25
[…] Also, since November 2021, Google has enforced new storage access limitations for apps published on its Play store which prohibits direct storage access on Android 11 and above forcing QField to adapt and rely on importing projects and datasets to access those. As part of the enforcement of these new policies, Google came up with an arbitrary mechanism to whitelist some apps which allows those to retain full storage access given the user explicitly allowed for it. We here at OPENGIS.ch believe QField had ample justifications to be whitelisted, however, Google’s appeal process judged otherwise after a series of email exchanges detailing our reasoning. While we have so far lost this argument with Google, we will continue fighting for our users and for their freedom to choose. If you are interested in more details, read our blog post about it. […]
QField Änderung Speicher - Baumsicht · 2022-05-05 at 05:47
[…] Ab Version 2 von QField mit Android 11 oder höher werden die Daten in einem anderen Verzeichnis gespeichert, siehe https://docs.qfield.org/get-started/storage/ und https://www.opengis.ch/2022/03/05/qfield-users-sit-down-we-need-to-talk-about-storage-access-on-andr… […]