Page 1 of 1
Forum

Welcome to the Tweaking4All community forums!
When participating, please keep the Forum Rules in mind!

Topics for particular software or systems: Start your topic link with the name of the application or system.
For example “MacOS X – Your question“, or “MS Word – Your Tip or Trick“.

Please note that switching to another language when reading a post will not bring you to the same post, in Dutch, as there is no translation for that post!



Share:
Notifications
Clear all

Huge output files

25 Posts
2 Users
0 Likes
439 Views
 Anonymous
Joined: 53 years ago
Posts: 0
Topic starter  

As described in my comments on your download page, I am getting massive output files. My drive is 30GB, and the output img file is 30GB uncompressed, despite having selected the shrink option. The process takes about an hour and a half.

Compressing it to a .tar.gz or a .zip results in a 17.5GB output file.

I followed your suggestion about resizing the partition to fill the drive. It made no difference. What can we do about this?

This topic was modified 5 months ago by

ReplyQuote
Topic Tags
 Hans
(@hans)
Famed Member Admin
Joined: 9 years ago
Posts: 2236
 

Hi Stuart! 😊 

Let's get this issue fixed 😁 

You resized the EXT (Linux) partition on your Pi, right?

Can you get the partition details in ApplePi-Baker?
I'm sure you have already seen and done this: just wondering what happened to the SD card after you resized.

Should look something like this:

Partition Table Info of /dev/disk5

  Size: 32,023,511,040 Bytes (62,545,920 Sectors)

  Partitioning Layout: Master Boot Record (MBR)

  Partition 1: FAT32 Partition
  - Start Sector   = 8,192 (4,194,304 Bytes)
  - Partition Size = 1,048,576 (536,870,912 Bytes)

  Partition 2: Linux native Partition
  - Start Sector   = 1,056,768 (541,065,216 Bytes)
  - Partition Size = 65,536 (33,554,432 Bytes)

  Partition 3: Empty Partition

  Partition 4: Empty Partition

 

 


ReplyQuote
 Anonymous
Joined: 53 years ago
Posts: 0
Topic starter  

No, there is nothing listed in that panel. Which probably explains why it's producing such large files, come to think of it.


ReplyQuote
 Hans
(@hans)
Famed Member Admin
Joined: 9 years ago
Posts: 2236
 

That will not be related to the file size ... it is weird though.

Another thing that is odd though:
How did you select the disk when there are no disks listed?
Does it show in the list when you click "Select a disk"?

When I have no suitable disks available, I do not get it in my popup:

And when I select a disk, and remove the disk after that, it is being removed under "Select a Disk" as well.

Also: to see all disks, click the disk icon in the upper right corner - let's see if it lists /dev.disk2 now ...

 

After that, something else to test: add an additional drive (eg. USB stick or external drive if you have one).


ReplyQuote
 Hans
(@hans)
Famed Member Admin
Joined: 9 years ago
Posts: 2236
 

p.s. enabling the log window may show some helpful info as well:


ReplyQuote
 Anonymous
Joined: 53 years ago
Posts: 0
Topic starter  

Well...if the program can't find any partitions, then presumably it will just make a raw backup, right? Hence a large file size.



ReplyQuote
 Hans
(@hans)
Famed Member Admin
Joined: 9 years ago
Posts: 2236
 

That's odd! Definitely some kind of bug we're looking at and I've got no clue (yet) why the list is empty on your Mac 🤔 
This doesn't seem to hinder APB from allowing you to select a drive - which is even weirder 😯 

Maybe this is caused by El Capitan ...?
The enhanced list is based on specific system calls, and maybe El Capitan doesn't support the way I'm using them?

Since it does make a backup though, let's see what we can try ...

Can you do a Terminal dump of your disks (the SD card inserted of course);

In Terminal:

diskutil list

 

Which should produce something like this, with specifics for your system of course. I'd only need to copy the relevant part, which in my example would be /dev/disk2.

/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk0
   1:                        EFI ⁨EFI⁩                     209.7 MB   disk0s1
   2:                 Apple_APFS ⁨Container disk1⁩         744.0 GB   disk0s2
   3:       Microsoft Basic Data ⁨BOOTCAMP⁩                256.3 GB   disk0s3

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +744.0 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume ⁨Macintosh HD - Data⁩     548.4 GB   disk1s1
   2:                APFS Volume ⁨Preboot⁩                 676.7 MB   disk1s2
   3:                APFS Volume ⁨Recovery⁩                1.1 GB     disk1s3
   4:                APFS Volume ⁨VM⁩                      2.1 GB     disk1s4
   5:                APFS Volume ⁨Macintosh HD⁩            15.8 GB    disk1s5
   6:              APFS Snapshot ⁨com.apple.os.update-...⁩ 15.8 GB    disk1s5s1

/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *32.0 GB    disk2
   1:             Windows_FAT_32 ⁨LIBREELEC⁩               536.9 MB   disk2s1
   2:                      Linux ⁨STORAGE⁩                 33.6 MB    disk2s2
                    (free space)                         31.4 GB    -

 

As you can see here: FDisk_partition_scheme (MBR), with a Windows_FAT_32 partition and a Linux partition.

What I'm looking for is how much of the disk space is actually being used.

 

To be clear: APB will first make a FULL backup. So this results in a 32 Gb IMG file (this does take quite some time indeed).
Next it will determine what the 1st Linux Partition is (assuming MBR and primary partitions - if you have an SSD then this should be fast).
The partition will then be extracted, resized, verified and reinserted (in general this is quite fast as well).
If compression is enabled (7z, ZIP, bz, xy, etc) then the new IMG file be compressed after all that (this again may take some time, depending on the chosen compression - 7z not being the best choice).

So the process will take at least as much time as backing up a plain 32Gb disk.


ReplyQuote
 Anonymous
Joined: 53 years ago
Posts: 0
Topic starter  

Diskutil prints:

/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *8.0 GB disk2
1: EFI EFISYS 819.2 KB disk2s1
2: 83BD6B9D-7F41-11DC-BE0B-001560B84F0F 59.4 KB disk2s2
3: FreeBSD Swap 524.3 KB disk2s3
4: FreeBSD UFS 1.9 GB disk2s4
/dev/disk3 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *30.8 GB disk3
1: Windows_FAT_32 boot 268.4 MB disk3s1
2: Linux 30.5 GB disk3s2

(Note that the BSD drive was only there because you asked me to add another flash drive.)


ReplyQuote
 Hans
(@hans)
Famed Member Admin
Joined: 9 years ago
Posts: 2236
 

That is unfortunate ... everything looks perfectly fine. 😣 

Do you have access to a Mac that runs a more recent macOS - just to test and see if this is an issue with your SD card or your system?

 

p.s.

Since you have a full backup already (with Linux partition maximized), we can do some test with that, which will be MUCH faster.
Maybe we can learn something here from the debug info without you having to run a 1-3 hour backup 😉 

1. Remove SD cards and USB disks - to avoid confusion.

2. Start ApplePi-Baker - you should not be able to select any disks.

3. Double click the IMG file you made earlier (unzipped 32Gb IMG).

This will make it that macOS will try to mount the disks. Normally only the boot (FAT32) partition would show in Finder.
If you have a tool like Paragon ExtFS installed, then the Linux partition will be mounted as well.

Apple Pi Baker will report that these have been mounted, and you should  be able to select the mounted disk now.

4. Enable the shrink option in APB, and start the backup.

Not sure if you run off an SSD or HDD, and how fast your computer is, but my Mac make a 4Gb backup (incl resizing) in 31 seconds.

Now look in the log Window, where you should see something like this (I think you may have looked at it before).
You can right click the log window and select "Copy Log to Clipboard" and paste it in a reply.

16:19:51  Backup -- START BACKUP
16:19:57  Backup Started - Making Disk to File Backup
16:19:57  Source - Disk: /dev/disk2 (4 GB Apple Disk Image) (3,623,878,656 bytes)
16:19:57  Destination - File: /Users/hans/Downloads/ApplePiBakerTests/backup.img
16:19:57  Shrink IMG Enabled - Temporary IMG file: "backup.tmp"
16:19:58  Backup started
16:20:05  Completed - Cloning Completed
16:20:05  Finished - Completed in 7 seconds, average speed 514.7 MB/sec
16:20:05  Resizing - Attempting IMG shrinking
16:20:05  RESIZE - Attempting to MINIMIZE a Linux partition in the IMG file "backup.tmp"
16:20:05  RESIZE - Find Ext2/3/4 partition and determine size and location
16:20:05  RESIZE - STATS:
16:20:05    File Size         : 3,623,878,656 Bytes (7,077,888 Sectors)
16:20:05    Linux Partition   : 3,351,248,896 Bytes (6,545,408 Sectors)
16:20:05    Starts at         : Byte 272,629,760 (Sector 532,480)
16:20:05    Linux Target Size : Minimum size
16:20:05  RESIZE - Extracting Ext2/3/4 Linux partition
16:20:12  RESIZE - Verifying and Fixing extracted Linux Partition
16:20:12  RESIZE - Shrinking Linux Partition to Minimum Size (double pass)
16:20:13  RESIZE - Creating new IMG file and Updating Partition Table IMG file
16:20:21  RESIZE - Shrinking of IMG file completed
16:20:21    Original:
16:20:21    - IMG File Size   : 3,623,878,656 Bytes
16:20:21    - Linux Partition : 3,351,248,896 Bytes
16:20:21    Resized:
16:20:21    - IMG File Size   : 3,623,878,656 Bytes
16:20:21    - Linux Partition : 3,351,248,896 Bytes
16:20:21    Bummer ... File size did not change in size by resizing the partition
16:20:22  Backup - Backup and Resizing Completed in 24 seconds
16:20:22  Backup -- END BACKUP

 

 


ReplyQuote
 Anonymous
Joined: 53 years ago
Posts: 0
Topic starter  

15:54:47 Initialization Info
15:54:47 - ApplePi-Baker 2.3.0 (Build 2)
15:54:47 - macOS Version 10.11.6 (Build 15G31) x86-64 (64 bit application)
15:54:47 - libarchive 3.6.0, by Tim Kientzle
15:54:47 - liblzma 5.2.5, by Mike Kezner et al.
15:54:47 - zlib 1.2.11, by Greg Roelofs && Mark Adler
15:54:47 - bz2lib 1.0.8, by Julian Seward
15:54:47 - e2fsprogs 1.46.5, by Theodore Ts'o
15:54:47 Found Correct HelperTool version (1.9.2)
15:54:47 Disk Appeared - /dev/disk2
15:54:47 Disk Appeared - /dev/disk1
15:54:47 Disk Appeared - /dev/disk0
15:54:52 Full Disk Access Test (1): FAILED
15:54:53 Full Disk Access Test (2): FAILED
15:54:54 Full Disk Access Test (3): FAILED
15:54:55 Full Disk Access Test (4): FAILED
15:54:56 Full Disk Access Test (5): FAILED
15:54:57 Full Disk Access Test (6): FAILED
15:54:58 Full Disk Access Test (7): FAILED
15:55:00 Full Disk Access Test (8): FAILED
15:55:01 Full Disk Access Test (9): FAILED
15:55:02 Full Disk Access Test (10): FAILED
15:55:03 Full Disk Access Test (11): FAILED
15:55:04 Full Disk Access Test (12): FAILED
15:55:05 Full Disk Access Test (13): FAILED
15:55:06 Full Disk Access Test (14): SUCCESS
15:55:06 Drive added to list: /dev/disk2 (31 GB) Mounted Image: "backup.img"

---

Yeah....not particularly fast ;)

This post was modified 5 months ago by

ReplyQuote
 Hans
(@hans)
Famed Member Admin
Joined: 9 years ago
Posts: 2236
 

Ehm, nope, not fast ... then again; considering you're still running El Capitan, I didn't expect it to be a speed monster but 1.1 MB/sec is a little slow indeed ... 
I sympathize with your suffering! 😉 


ReplyQuote
 Anonymous
Joined: 53 years ago
Posts: 0
Topic starter  

And no, I don't have another Mac. Eventually I will upgrade this one's OS, but not today. Something might break.

Maybe if you compiled a version with extra debug logging or something, and I ran that?


ReplyQuote
 Hans
(@hans)
Famed Member Admin
Joined: 9 years ago
Posts: 2236
 
Posted by: @sodalmighty

Something might break

I can totally get that - you have no idea how much time it has cost me, as a developer, to keep things working because Apple changed something again (PowerPC to Intel, required signing apps with a $99/year CERT, changes in security (user cannot erase a disk anymore in regular apps), notarizing apps, forcing everything 64 bit, and now M1) ... 😞 

Compiling a special debug version would include a ton of extra info and would require to have at least lldb (debugger) and my dev environment to run. So I don't really consider that an option.
I do have some debug statements in my own code, but they won't tell us what is going wrong.

The bad part about this is that even if I would have a copy of the IMG file, then it still may work just fine on my Mac. That's why I asked if you would have access to do this test on another Mac.
Emailing a 17Gb file still doesn't work that well 🤣  

So one thing we can do is trying to resize the Linux partition manually, but for that we need to find a way to extract just the Linux partition from the IMG file.
For now I do not know an easy way to do this. I'd suggest quickly copying the extracted Linux Partition file, but seeing how slow your disk is, I'd give it a good chance you won't be able to catch that correctly.

Can you try if the resize tools work properly?

In Terminal, you can run the resize (resize2fs) and file system check (e2fsck) tools like so.
Do you see this kind of output?

$ cd /Applications/ApplePiBaker.app/Contents/MacOS

$ ./resize2fs 

resize2fs 1.46.5 (30-Dec-2021)
Usage: ./resize2fs [-d debug_flags] [-f] [-F] [-M] [-P] [-p] device [-b|-s|new_size] [-S RAID-stride] [-z undo_file]

$ ./e2fsck

Usage: ./e2fsck [-panyrcdfktvDFV] [-b superblock] [-B blocksize]
		[-l|-L bad_blocks_file] [-C fd] [-j external_journal]
		[-E extended-options] [-z undo_file] device

Emergency help:
 -p                   Automatic repair (no questions)
 -n                   Make no changes to the filesystem
 -y                   Assume "yes" to all questions
 -c                   Check for bad blocks and add them to the badblock list
 -f                   Force checking even if filesystem is marked clean
 -v                   Be verbose
 -b superblock        Use alternative superblock
 -B blocksize         Force blocksize when looking for superblock
 -j external_journal  Set location of the external journal
 -l bad_blocks_file   Add to badblocks list
 -L bad_blocks_file   Set badblocks list
 -z undo_file         Create an undo file

ReplyQuote
 Anonymous
Joined: 53 years ago
Posts: 0
Topic starter  

Aha! We may be onto something here.

% ./e2fsck
dyld: lazy symbol binding failed: Symbol not found: ____chkstk_darwin
Referenced from: /Applications/ApplePiBaker.app/Contents/MacOS/./e2fsck
Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ____chkstk_darwin
Referenced from: /Applications/ApplePiBaker.app/Contents/MacOS/./e2fsck
Expected in: /usr/lib/libSystem.B.dylib

zsh: trace trap ./e2fsck


ReplyQuote
 Hans
(@hans)
Famed Member Admin
Joined: 9 years ago
Posts: 2236
 

Yep, that would explain why resizing doesn't work ... yikes.

I did recompile both tools, just in case I goofed up, enforcing macOS 10.8 (should work then for El Capitan for sure right?).
Unfortunately, the oldest macOS I have is 10.13, and under 10.13 both versions of the tools work (the ones that came with your APB and the ones I just recompiled) - so it is not giving me an answer.

Would you be willing to test these?
They are not signed or notarized - so security may be complaining on you Mac.
You can download both command line tools here: link.
After unzipping, run them from Terminal.

Since I'm not familiar with your Terminal skills, I'll try to write down every step - apologies if this is already something you're familiar with ...

  1. download this file for example to your Downloads folder.
  2. double click the file, which will create a directory called "macos-extfs-resize-tools-recompiled" in your "Download" folder, and both files will be extracted in that folder.
  3. Open Terminal (Applications - Utilities - Terminal) and execute these commands:
    cd ~/Downloads/macos-extfs-resize-tools-recompiled
    
    ./resize2fs
    
    ./e2fsck

    If all went well the output should look something like this:

    cd ~/Downloads/macos-extfs-resize-tools-recompiled
    
    $ ./resize2fs
    
    resize2fs 1.46.5 (30-Dec-2021)
    Usage: ./resize2fs [-d debug_flags] [-f] [-F] [-M] [-P] [-p] device [-b|-s|new_size] [-S RAID-stride] [-z undo_file]
    
    $ ./e2fsck
    
    Usage: ./e2fsck [-panyrcdfktvDFV] [-b superblock] [-B blocksize]
    		[-l|-L bad_blocks_file] [-C fd] [-j external_journal]
    		[-E extended-options] [-z undo_file] device
    
    Emergency help:
     -p                   Automatic repair (no questions)
     -n                   Make no changes to the filesystem
     -y                   Assume "yes" to all questions
     -c                   Check for bad blocks and add them to the badblock list
     -f                   Force checking even if filesystem is marked clean
     -v                   Be verbose
     -b superblock        Use alternative superblock
     -B blocksize         Force blocksize when looking for superblock
     -j external_journal  Set location of the external journal
     -l bad_blocks_file   Add to badblocks list
     -L bad_blocks_file   Set badblocks list
     -z undo_file         Create an undo file

ReplyQuote
Page 1 / 2
Share: