Full local backup infrastructure

This is the basic infrastructure for pulling a full(*) backup of the
device's data over an adb(**) connection to the local device.  The
basic process consists of these interacting pieces:

1. The framework's BackupManagerService, which coordinates the
   collection of app data and routing to the destination.

2. A new framework-provided BackupAgent implementation called
   FullBackupAgent, which is instantiated in the target applications'
   processes in turn, and knows how to emit a datastream that contains
   all of the app's saved data files.

3. A new shell-level program called "bu" that is used to bridge from
   adb to the framework's Backup Manager.

4. adb itself, which now knows how to use 'bu' to kick off a backup
   operation and pull the resulting data stream to the desktop host.

5. A system-provided application that verifies with the user that
   an attempted backup/restore operation is in fact expected and to
   be allowed.

The full agent implementation is not used during normal operation of
the delta-based app-customized remote backup process.  Instead it's
used during user-confirmed *full* backup of applications and all their
data to a local destination, e.g. via the adb connection.

The output format is 'tar'.  This makes it very easy for the end
user to examine the resulting dataset, e.g. for purpose of extracting
files for debug purposes; as well as making it easy to contemplate
adding things like a direct gzip stage to the data pipeline during
backup/restore.  It also makes it convenient to construct and maintain
synthetic backup datasets for testing purposes.

Within the tar format, certain artificial conventions are used.
All files are stored within top-level directories according to
their semantic origin:

apps/pkgname/a/  : Application .apk file itself
apps/pkgname/obb/: The application's associated .obb containers
apps/pkgname/f/  : The subtree rooted at the getFilesDir() location
apps/pkgname/db/ : The subtree rooted at the getDatabasePath() parent
apps/pkgname/sp/ : The subtree rooted at the getSharedPrefsFile() parent
apps/pkgname/r/  : Files stored relative to the root of the app's file tree
apps/pkgname/c/  : Reserved for the app's getCacheDir() tree; not stored.

For each package, the first entry in the tar stream is a file called
"_manifest", nominally rooted at apps/pkgname.  This file contains some
metadata about the package whose data is stored in the archive.

The contents of shared storage can optionally be included in the tar
stream. It is placed in the synthetic location:

shared/...

uid/gid are ignored; app uids are assigned at install time, and the
app's data is handled from within its own execution environment, so
will automatically have the app's correct uid.

Forward-locked .apk files are never backed up.  System-partition
.apk files are not backed up unless they have been overridden by a
post-factory upgrade, in which case the current .apk *is* backed up --
i.e. the .apk that matches the on-disk data.  The manifest preceding
each application's portion of the tar stream provides version numbers
and signature blocks for version checking, as well as an indication
of whether the restore logic should expect to install the .apk before
extracting the data.

System packages can designate their own full backup agents.  This is
to manage things like the settings provider which (a) cannot be shut
down on the fly in order to do a clean snapshot of their file trees,
and (b) manage data that is not only irrelevant but actively hostile
to non-identical devices -- CDMA telephony settings would seriously
mess up a GSM device if emplaced there blind, for example.

When a full backup or restore is initiated from adb, the system will
present a confirmation UI that the user must explicitly respond to
within a short [~ 30 seconds] timeout.  This is to avoid the
possibility of malicious desktop-side software secretly grabbing a copy
of all the user's data for nefarious purposes.

(*) The backup is not strictly a full mirror.  In particular, the
    settings database is not cloned; it is handled the same way that
    it is in cloud backup/restore.  This is because some settings
    are actively destructive if cloned onto a different (or
    especially a different-model) device: telephony settings and
    AndroidID are good examples of this.

(**) On the framework side it doesn't care that it's adb; it just
    sends the tar stream to a file descriptor.  This can easily be
    retargeted around whatever transport we might decide to use
    in the future.

KNOWN ISSUES:

* the security UI is desperately ugly; no proper designs have yet
  been done for it
* restore is not yet implemented
* shared storage backup is not yet implemented
* symlinks aren't yet handled, though some infrastructure for
  dealing with them has been put in place.

Change-Id: Ia8347611e23b398af36ea22c36dff0a276b1ce91

3 files changed, 204 insertions, 3 deletions

diff --git a/include/utils/BackupHelpers.h b/include/utils/BackupHelpers.h
index b1f5045..1bb04a7 100644
--- a/include/utils/BackupHelpers.h
+++ b/include/utils/BackupHelpers.h
@@ -70,6 +70,14 @@ public:
     ~BackupDataWriter();
 
     status_t WriteEntityHeader(const String8& key, size_t dataSize);
+
+    /* Note: WriteEntityData will write arbitrary data into the file without
+     * validation or a previously-supplied header.  The full backup implementation
+     * uses it this way to generate a controlled binary stream that is not
+     * entity-structured.  If the implementation here is changed, either this
+     * use case must remain valid, or the full backup implementation should be
+     * adjusted to use some other appropriate mechanism.
+     */
     status_t WriteEntityData(const void* data, size_t size);
 
     void SetKeyPrefix(const String8& keyPrefix);
@@ -103,7 +111,7 @@ public:
 
     bool HasEntities();
     status_t ReadEntityHeader(String8* key, size_t* dataSize);
-    status_t SkipEntityData(); // must be called with the pointer at the begining of the data.
+    status_t SkipEntityData(); // must be called with the pointer at the beginning of the data.
     ssize_t ReadEntityData(void* data, size_t size);
 
 private:
@@ -126,6 +134,9 @@ private:
 int back_up_files(int oldSnapshotFD, BackupDataWriter* dataStream, int newSnapshotFD,
         char const* const* files, char const* const *keys, int fileCount);
 
+int write_tarfile(const String8& packageName, const String8& domain,
+        const String8& rootPath, const String8& filePath, BackupDataWriter* outputStream);
+
 class RestoreHelperBase
 {
 public:
diff --git a/libs/utils/BackupData.cpp b/libs/utils/BackupData.cpp
index adb3174..f963058 100644
--- a/libs/utils/BackupData.cpp
+++ b/libs/utils/BackupData.cpp
@@ -20,12 +20,15 @@
 #include <utils/ByteOrder.h>
 
 #include <stdio.h>
+#include <string.h>
 #include <unistd.h>
 
 #include <cutils/log.h>
 
 namespace android {
 
+static const bool DEBUG = false;
+
 /*
  * File Format (v1):
  *
@@ -75,6 +78,7 @@ BackupDataWriter::write_padding_for(int n)
     paddingSize = padding_extra(n);
     if (paddingSize > 0) {
         uint32_t padding = 0xbcbcbcbc;
+        if (DEBUG) LOGI("writing %d padding bytes for %d", paddingSize, n);
         amt = write(m_fd, &padding, paddingSize);
         if (amt != paddingSize) {
             m_status = errno;
@@ -107,8 +111,8 @@ BackupDataWriter::WriteEntityHeader(const String8& key, size_t dataSize)
     } else {
         k = key;
     }
-    if (false) {
-        LOGD("Writing entity: prefix='%s' key='%s' dataSize=%d", m_keyPrefix.string(), key.string(),
+    if (DEBUG) {
+        LOGD("Writing header: prefix='%s' key='%s' dataSize=%d", m_keyPrefix.string(), key.string(),
                 dataSize);
     }
 
@@ -121,6 +125,7 @@ BackupDataWriter::WriteEntityHeader(const String8& key, size_t dataSize)
     header.keyLen = tolel(keyLen);
     header.dataSize = tolel(dataSize);
 
+    if (DEBUG) LOGI("writing entity header, %d bytes", sizeof(entity_header_v1));
     amt = write(m_fd, &header, sizeof(entity_header_v1));
     if (amt != sizeof(entity_header_v1)) {
         m_status = errno;
@@ -128,6 +133,7 @@ BackupDataWriter::WriteEntityHeader(const String8& key, size_t dataSize)
     }
     m_pos += amt;
 
+    if (DEBUG) LOGI("writing entity header key, %d bytes", keyLen+1);
     amt = write(m_fd, k.string(), keyLen+1);
     if (amt != keyLen+1) {
         m_status = errno;
@@ -145,7 +151,12 @@ BackupDataWriter::WriteEntityHeader(const String8& key, size_t dataSize)
 status_t
 BackupDataWriter::WriteEntityData(const void* data, size_t size)
 {
+    if (DEBUG) LOGD("Writing data: size=%lu", (unsigned long) size);
+
     if (m_status != NO_ERROR) {
+        if (DEBUG) {
+            LOGD("Not writing data - stream in error state %d (%s)", m_status, strerror(m_status));
+        }
         return m_status;
     }
 
@@ -155,6 +166,7 @@ BackupDataWriter::WriteEntityData(const void* data, size_t size)
     ssize_t amt = write(m_fd, data, size);
     if (amt != (ssize_t)size) {
         m_status = errno;
+        if (DEBUG) LOGD("write returned error %d (%s)", m_status, strerror(m_status));
         return m_status;
     }
     m_pos += amt;
diff --git a/libs/utils/BackupHelpers.cpp b/libs/utils/BackupHelpers.cpp
index 4ad9b51..ad4a308 100644
--- a/libs/utils/BackupHelpers.cpp
+++ b/libs/utils/BackupHelpers.cpp
@@ -442,6 +442,184 @@ back_up_files(int oldSnapshotFD, BackupDataWriter* dataStream, int newSnapshotFD
     return 0;
 }
 
+// Utility function, equivalent to stpcpy(): perform a strcpy, but instead of
+// returning the initial dest, return a pointer to the trailing NUL.
+static char* strcpy_ptr(char* dest, const char* str) {
+    if (dest && str) {
+        while ((*dest = *str) != 0) {
+            dest++;
+            str++;
+        }
+    }
+    return dest;
+}
+
+int write_tarfile(const String8& packageName, const String8& domain,
+        const String8& rootpath, const String8& filepath, BackupDataWriter* writer)
+{
+    // In the output stream everything is stored relative to the root
+    const char* relstart = filepath.string() + rootpath.length();
+    if (*relstart == '/') relstart++;     // won't be true when path == rootpath
+    String8 relpath(relstart);
+
+    // Too long a name for the ustar format?
+    //    "apps/" + packagename + '/' + domainpath < 155 chars
+    //    relpath < 100 chars
+    if ((5 + packageName.length() + 1 + domain.length() >= 155) || (relpath.length() >= 100)) {
+        LOGE("Filename [%s] too long, skipping", relpath.string());
+        return -1;
+    }
+
+    int err = 0;
+    struct stat64 s;
+    if (lstat64(filepath.string(), &s) != 0) {
+        err = errno;
+        LOGE("Error %d (%s) from lstat64(%s)", err, strerror(err), filepath.string());
+        return err;
+    }
+
+    const int isdir = S_ISDIR(s.st_mode);
+
+    // !!! TODO: use mmap when possible to avoid churning the buffer cache
+    // !!! TODO: this will break with symlinks; need to use readlink(2)
+    int fd = open(filepath.string(), O_RDONLY);
+    if (fd < 0) {
+        err = errno;
+        LOGE("Error %d (%s) from open(%s)", err, strerror(err), filepath.string());
+        return err;
+    }
+
+    // read/write up to this much at a time.
+    const size_t BUFSIZE = 32 * 1024;
+
+    char* buf = new char[BUFSIZE];
+    if (buf == NULL) {
+        LOGE("Out of mem allocating transfer buffer");
+        err = ENOMEM;
+        goto done;
+    }
+
+    // Good to go -- first construct the standard tar header at the start of the buffer
+    memset(buf, 0, 512);    // tar header is 512 bytes
+
+    // Magic fields for the ustar file format
+    strcat(buf + 257, "ustar");
+    strcat(buf + 263, "00");
+
+    {
+        // Prefix and main relative path.  Path lengths have been preflighted.
+
+        // [ 345 : 155 ] filename path prefix [ustar]
+        //
+        // packagename and domain can each be empty.
+        char* cp = buf + 345;
+        if (packageName.length() > 0) {
+            // it's an app; so prefix with "apps/packagename/"
+            cp = strcpy_ptr(cp, "apps/");
+            cp = strcpy_ptr(cp, packageName.string());
+        }
+
+        if (domain.length() > 0) {
+            // only need a / if there was a package name
+            if (packageName.length() > 0) *cp++ = '/';
+            cp = strcpy_ptr(cp, domain.string());
+        }
+
+        // [   0 : 100 ]; file name/path
+        strncpy(buf, relpath.string(), 100);
+
+        LOGI("   Name: %s/%s", buf + 345, buf);
+    }
+
+    // [ 100 :   8 ] file mode
+    snprintf(buf + 100, 8, "0%o", s.st_mode);
+
+    // [ 108 :   8 ] uid -- ignored in Android format; uids are remapped at restore time
+    // [ 116 :   8 ] gid -- ignored in Android format
+    snprintf(buf + 108, 8, "0%lo", s.st_uid);
+    snprintf(buf + 116, 8, "0%lo", s.st_gid);
+
+    // [ 124 :  12 ] file size in bytes
+    snprintf(buf + 124, 12, "0%llo", s.st_size);
+
+    // [ 136 :  12 ] last mod time as a UTC time_t
+    snprintf(buf + 136, 12, "%0lo", s.st_mtime);
+
+    // [ 148 :   8 ] checksum -- to be calculated with this field as space chars
+    memset(buf + 148, ' ', 8);
+
+    // [ 156 :   1 ] link/file type
+    uint8_t type;
+    if (isdir) {
+        type = '5';     // tar magic: '5' == directory
+    } else if (S_ISREG(s.st_mode)) {
+        type = '0';     // tar magic: '0' == normal file
+    } else {
+        LOGW("Error: unknown file mode 0%o [%s]", s.st_mode, filepath.string());
+        goto cleanup;
+    }
+    buf[156] = type;
+
+    // [ 157 : 100 ] name of linked file [not implemented]
+
+    // Now go back and calculate the header checksum
+    {
+        uint16_t sum = 0;
+        for (uint8_t* p = (uint8_t*) buf; p < ((uint8_t*)buf) + 512; p++) {
+            sum += *p;
+        }
+
+        // Now write the real checksum value:
+        // [ 148 :   8 ]  checksum: 6 octal digits [leading zeroes], NUL, SPC
+        sprintf(buf + 148, "%06o", sum); // the trailing space is already in place
+    }
+
+    // Write the 512-byte tar file header block to the output
+    writer->WriteEntityData(buf, 512);
+
+    // Now write the file data itself, for real files.  We honor tar's convention that
+    // only full 512-byte blocks are sent to write().
+    if (!isdir) {
+        off64_t toWrite = s.st_size;
+        while (toWrite > 0) {
+            size_t toRead = (toWrite < BUFSIZE) ? toWrite : BUFSIZE;
+            ssize_t nRead = read(fd, buf, toRead);
+            if (nRead < 0) {
+                err = errno;
+                LOGE("Unable to read file [%s], err=%d (%s)", filepath.string(),
+                        err, strerror(err));
+                break;
+            } else if (nRead == 0) {
+                LOGE("EOF but expect %lld more bytes in [%s]", (long long) toWrite,
+                        filepath.string());
+                err = EIO;
+                break;
+            }
+
+            // At EOF we might have a short block; NUL-pad that to a 512-byte multiple.  This
+            // depends on the OS guarantee that for ordinary files, read() will never return
+            // less than the number of bytes requested.
+            ssize_t partial = (nRead+512) % 512;
+            if (partial > 0) {
+                ssize_t remainder = 512 - partial;
+                memset(buf + nRead, 0, remainder);
+                nRead += remainder;
+            }
+            writer->WriteEntityData(buf, nRead);
+            toWrite -= nRead;
+        }
+    }
+
+cleanup:
+    delete [] buf;
+done:
+    close(fd);
+    return err;
+}
+// end tarfile
+
+
+
 #define RESTORE_BUF_SIZE (8*1024)
 
 RestoreHelperBase::RestoreHelperBase()