Table of contents

Preliminary note: this post was initially published on LinuxFr.org where readers have provided many good comments. If you ever end up in the same situation as described below have a look at tools like sgdisk, pvresize, lvresize, pvmove.

I was minding my own business when suddenly there wasn’t so much free space on the household’s NAS. We’ve been trying to scrap a gigabyte here and there since months and the situation wasn’t going any better. 500 GB filled to the top and nothing left to remove.

I give in, let’s order a 1 TB SSD to replace the 500 GB one, and while I’m at it I will change the remaining mechanical hard drives from the other computers in the house.

dd

The NAS is actually an old Eee PC with a hard drive quite difficult to remove. One has to take the keyboard apart, then to unscrew and to remove the plastic cover, and finally to unplug a flat cable. Only then one can extract the support on which the hard drive is screwed. While I was at it I took the opportunity to break the flat cable’s plug, not without being quite astonished at how much strength was needed to unplug it. Well, it seems that the correct amount of force to apply was zero, just like the previous eight thousands times I broke something by forcing… This is a lesson that I can’t remember apparently.

Thankfully I managed to put the cable back in place. the small cable for the touchpad, on the other hand, won’t stay in its cracked slot, relic of an old time when I used to apply force while disassembling computers. Fortunately this time is over. Anyway, who cares, does anyone need a touchpad on a NAS?

I’ve put the old disk out and since I did not want to do an install from scratch I thought, how clever, that I could simply transfer the data from the old disk to the new one with the help of dd. I’m such a haxor, so smart. It happens that it actually was the worst idea.

With both disks in USB cases and plugged to another computer, I launch dd. It takes 5 hours with an average speed of 30 MB/s, with no surprise. Practical USB is far from the theoretical performance.

Once the copy is over I launch GParted. It outputs a message that I don’t understand, about a GPT backup, and tells me that it can solve the problem. Well, good idea Mr. software, just do that. The it correctly displays the three partitions: GPT, /boot, and /; the last one being encrypted. I extend / to use all the available space then I install the disk into the NAS.

The systems starts without any issue. I type the password to decrypt the disk, it works, then the boot completes as expected. I type my login and my password, then I check that the whole operation as given the expected results.

$ df --human-readable
Filesystem             Size  Used  Avail  Capacity  Mounted on
/dev/sda2                1G  430M   570M       42%  /boot
/dev/mapper/ubuntu-vg  500G  498G     2M       99%  /

Wait, where is my tera ?

resize2fs

A run of fdisk --list shows that all partition have the expected size, the whole disk is occupied. The whole disk? Yes, in its entirety. I don’t know what is happening and my hypothesis is that there is a difference between the size as shown in the partition table and the one seen by ext4. Unless it comes from the LVM between them? Updated strategy:

$ resize2fs /dev/mapper/ubuntu-vg
resize2fs: New size results in too many block group descriptors.

Uh! Really, nothing works here! After a quick search on the web I find an internet user pretending I’m doing nonsense. Pfff, what does he know this tytso?

Well, okay, I swallow my pride… It’s been somewhere near eight hours that I am juggling with turds without knowing.

Device UUID

How well! I’m going to format the disk correctly and to transfer the data from the old one to the new one using rsync. Since I don’t want to work on the resource-limited NAS I extract the disk once again, put it back in the USB case, and plug everything on the other computer. I will learn afterwards that it actually was the worst idea.

On the other computer, it’s a mess. GParted can’t create the partitions. It keeps showing the same GPT error as above. I run fdisk but it also reports the GPT error. An internet user suggests to erase the partitions then to create them again with the same offsets, without erasing the data. It’s fun, and a bit scary, but it does not help.

Suddenly, I see the light. I read in the output of fdisk that both disks have the same UUID. I unplug one of them, and boom, no more GPT issue. It seems that the system loses its mind when two resources have the same Universally Unique Identifier. Who could have seen that coming?

After having changed the disk’s UUID with the help of the web (this knowledge base is really handy), I set the encrypted partition square. I mount it, everything seems fine. I plug the other disk, 🎵touc-ta🎵ta-touc🎵 (sounds of GNOME mounting and unmounting the disk. Well, well, well, I think to myself, it seems that there’s a bug in Nautilus. I don’t care, I don’t need the UI, I will mount the partition in a terminal.

mount: unknown filesystem type 'LVM2_member'

Coooome oooooooon! What’s the problem this time? Once again, a search on the web provides a quick answer (this web is definitely handy). It happens that each of these disks belong to a volume group, and that these groups have the same name. It becomes quite hard to follow for the system. What a good idea of having used dd!

rsync

I eventually rename the volume group, which allows me to mount both disks simultaneously. One rsync --archive and six hours later I can put the disk back in the NAS. It boots, then asks for the disk password, as expected.

Not enough available memory to open a keyslot.

Well okay, I must be located on an Indian burial ground or something like that. A new search on the web (this tool is quite handy), and I learn that the algorithm applied by Luks depends on the system’s resources, thus it obviously used the characteristics from the computer on which I did all the operations described above, a computer far more powerful than the Eee PC. The computer where I had plugged the disks. To. Save. Time.

grub-install

Let’s not lose the good mood. I boot the NAS on a live distribution to redo the whole partitioning. No problem here except that the menus in the live Ubuntu Server use a white font on yellow background. It’s unreadable but if I squint my eyes enough I manage to set my system up. Once the partition is ready I remove the disk from the NAS and plug it once again on the other computer. The disk is correctly listed and I can mount it. I can access both disks at once, I copy the data with rsync, then for the last step I just have to install GRUB. This part is honestly easy, I did it many times.

$ cryptsetup luksOpen /dev/sdb3
$ mount /dev/mapper/nas-vg /mnt
$ mount /dev/sdb2 /mnt/boot
$ mount --bind /dev /mnt/dev
$ mount --bind /proc /mnt/proc
$ mount --bind /sys /mnt/sys
$ chroot /mnt
$ grub-install /dev/sdb

This being done I put the disk back into the NAS, turn the computer on, aaaaaaaaaaaaand… no GRUB. Well, actually GRUB is here but tells me it doesn’t know what to do.

update-initramfs

After a quick search (you’ll guess where I found the information) I learn that I should have used update-initramfs -u -k all after the previous commands. No worries, I do that from the live system.

cryptsetup: WARNING: failed to detect canonical device of overlayfs
cryptsetup: WARNING: could not determine root device from /etc/fstab

Uh-oh, it doesn’t smell good again. After another round on the web I understand that there is an inconsistency between the partition UUIDs as shown by lsblk --fs and those written in /etc/fstab and /etc/cryptrab. What a good idea to have copied everything with rsync!

After having adapted the two last files with the output of lsblk I run update-initramfs which terminates with no error nor warning. I reboot the NAS, type the disk’s password… everything works well.

$ df --human-readable
Filesystem          Size  Used Avail Use% Mounted on
/dev/mapper/nas-vg  914G  434G  434G  51% /
/dev/sda2           2.0G  181M  1.7G  10% /boot

It was soooooooo easy.

Conclusion

We sometime think that we’ll save some time by being smart then eventually everything is a waste of time. We then attempt to save face by saying that we’ve leaned a lot.

We sometime do things even though we know quite well that this is a bad idea. We may even already have been burnt multiple times (RIP touchpad’s flat cable).

This story shows what can happen when we combine both: a sequence of bad ideas and decisions taken while being misinformed, all of it converging toward maximum wasted time. If I was paid for spent energy I would be rich.

For the other computers I chose a full reformat and system installation, followed by a rsync of $HOME. I encountered some minor issues but nothing as painful as all of this.

Huge thanks to all the internet users who take the time to explain their problem, and to all of those who take the time to suggest solutions. I hope that this marvelous space of sharing will stay this way and won’t be polluted by parasites.

At least I have learned a lot.