Add post about my Mastodon DB hickup
This commit is contained in:
parent
7d13a05f32
commit
4429b381d0
169
content/blog/mastodon-database-hickup.rst
Normal file
169
content/blog/mastodon-database-hickup.rst
Normal file
@ -0,0 +1,169 @@
|
|||||||
|
Mastodon database hickup
|
||||||
|
########################
|
||||||
|
|
||||||
|
:date: 2020-04-23T11:17Z
|
||||||
|
:category: blog
|
||||||
|
:tags: postgresql,mastodon
|
||||||
|
:url: blog/mastodon-database-hickup/
|
||||||
|
:save_as: blog/mastodon/database-hickup/index.html
|
||||||
|
:status: published
|
||||||
|
:author: Gergely Polonkai
|
||||||
|
|
||||||
|
|
||||||
|
After being online for almost a year, my `Mastodon instance <https://social.polonkai.eu>`_ is back
|
||||||
|
up. However, it wasn’t as straightforward as i wanted it to be.
|
||||||
|
|
||||||
|
About a year ago my Mastodon server went down. I messed up my docker images big time and the
|
||||||
|
then-new version 2.8.4 didn’t want to start up. I left it there, and as i had a pretty rough year
|
||||||
|
(in the good sense) i didn’t have time to bring it up again.
|
||||||
|
|
||||||
|
One of the constant problems i had with this instance is the size of the attachment directory. It
|
||||||
|
easily grew to an enormous size in no time, so i had to constantly clean things up. Plus, it took
|
||||||
|
away the precious space away from my other projects. So my first move was moving all these files
|
||||||
|
to a DigitalOcean Space; with the ``awscli`` Python package it was easy as pie. If you want to do
|
||||||
|
it, too, i followed `this article
|
||||||
|
<https://angristan.xyz/2018/05/moving-mastodon-media-files-to-wasabi-object-storage/>`_.
|
||||||
|
|
||||||
|
Next came the database.
|
||||||
|
|
||||||
|
I tried to load the database dump, created using ``pg_dump -f mastodon.sql mastodon``, with
|
||||||
|
``psql -f mastodon.sql``. Everything went fine, all the records created, except this error
|
||||||
|
message:
|
||||||
|
|
||||||
|
.. code-block:: text
|
||||||
|
|
||||||
|
psql:mastodon-dump.sql:1691661: ERROR: could not create unique index "index_tags_on_name"
|
||||||
|
DETAIL: Key (name)=(opensource) is duplicated.
|
||||||
|
|
||||||
|
First i tried to ignore it, but running `rails db:migrate` failed when it wanted to manipulate
|
||||||
|
this index.
|
||||||
|
|
||||||
|
So i issued this query:
|
||||||
|
|
||||||
|
.. code-block:: sql
|
||||||
|
|
||||||
|
SELECT COUNT(*) AS count, name FROM tags GROUP BY name HAVING COUNT(*) > 1;
|
||||||
|
|
||||||
|
And the result was this:
|
||||||
|
|
||||||
|
.. code-block:: text
|
||||||
|
|
||||||
|
count | name
|
||||||
|
2 | opensource
|
||||||
|
2 | xmpp
|
||||||
|
2 | fdroid
|
||||||
|
2 | socialmedia
|
||||||
|
2 | blogging
|
||||||
|
2 | c64
|
||||||
|
2 | blog
|
||||||
|
|
||||||
|
Then collected the IDs of these duplicate tag records using this query:
|
||||||
|
|
||||||
|
.. code-block:: sql
|
||||||
|
|
||||||
|
SELECT id, name, created_at
|
||||||
|
FROM tags
|
||||||
|
WHERE name in ('opensource',
|
||||||
|
'xmpp',
|
||||||
|
'fdroid',
|
||||||
|
'socialmedia',
|
||||||
|
'blogging',
|
||||||
|
'c64',
|
||||||
|
'blog')
|
||||||
|
ORDER BY name, created_at;
|
||||||
|
|
||||||
|
.. code-block:: text
|
||||||
|
|
||||||
|
id | name | created_at
|
||||||
|
-------+-------------+------------
|
||||||
|
159 | blog | 2018-02-24 19:56:01.710702
|
||||||
|
15941 | blog | 2019-04-03 22:05:56.438488
|
||||||
|
158 | blogging | 2018-02-24 19:56:01.721354
|
||||||
|
16006 | blogging | 2019-04-09 12:13:20.852976
|
||||||
|
5441 | c64 | 2018-07-31 00:50:56.172468
|
||||||
|
16036 | c64 | 2019-04-14 19:56:40.692197
|
||||||
|
924 | fdroid | 2018-04-14 19:39:21.261817
|
||||||
|
15947 | fdroid | 2019-04-04 19:10:50.190317
|
||||||
|
237 | opensource | 2018-03-05 21:50:52.609723
|
||||||
|
15929 | opensource | 2019-04-03 11:49:21.772961
|
||||||
|
251 | socialmedia | 2018-03-06 23:02:33.573775
|
||||||
|
16034 | socialmedia | 2019-04-14 19:07:37.081635
|
||||||
|
519 | xmpp | 2018-03-28 16:02:05.023784
|
||||||
|
15988 | xmpp | 2019-04-07 08:06:59.429965
|
||||||
|
|
||||||
|
I wanted to see the ``created_at`` field just to know when things got bad. As it turns out, it
|
||||||
|
was around April 2019, so it might have happened with the 2.8.0 upgrade. Was it me, or some
|
||||||
|
problem with the migrations remains a mistery.
|
||||||
|
|
||||||
|
I also wanted to know about all the tables referencing tags:
|
||||||
|
|
||||||
|
.. code-block:: sql
|
||||||
|
|
||||||
|
SELECT
|
||||||
|
tc.table_name, kcu.column_name
|
||||||
|
FROM
|
||||||
|
information_schema.table_constraints AS tc
|
||||||
|
JOIN information_schema.key_column_usage AS kcu
|
||||||
|
ON tc.constraint_name = kcu.constraint_name
|
||||||
|
JOIN information_schema.constraint_column_usage AS ccu
|
||||||
|
ON ccu.constraint_name = tc.constraint_name
|
||||||
|
WHERE constraint_type = 'FOREIGN KEY' AND ccu.table_name = 'tags';
|
||||||
|
|
||||||
|
.. code-block:: text
|
||||||
|
|
||||||
|
table_name | column_name
|
||||||
|
-------------------+-------------
|
||||||
|
statuses_tags | tag_id
|
||||||
|
account_tag_stats | tag_id
|
||||||
|
|
||||||
|
And last, but not least, i had to massage the dump a bit. After creating a backup, i opened up
|
||||||
|
the file in ViM and looked for the place where it was loading the data into the ``tags`` table.
|
||||||
|
It looked like this:
|
||||||
|
|
||||||
|
.. code-block:: sql
|
||||||
|
|
||||||
|
COPY public.tags (name, created_at, updated_at, id) FROM stdin;
|
||||||
|
blogging 2018-02-24 19:56:01.710702 2018-02-24 19:56:01.710702 158
|
||||||
|
blog 2018-02-24 19:56:01.721354 2018-02-24 19:56:01.721354 159
|
||||||
|
opensource 2018-03-05 21:50:52.609723 2018-03-05 21:50:52.609723 237
|
||||||
|
socialmedia 2018-03-06 23:02:33.573775 2018-03-06 23:02:33.573775 251
|
||||||
|
xmpp 2018-03-28 16:02:05.023784 2018-03-28 16:02:05.023784 519
|
||||||
|
fdroid 2018-04-14 19:39:21.261817 2018-04-14 19:39:21.261817 924
|
||||||
|
c64 2018-07-31 00:50:56.172468 2018-07-31 00:50:56.172468 5441
|
||||||
|
opensource 2019-04-03 11:49:21.772961 2019-04-03 11:49:21.772961 15929
|
||||||
|
blog 2019-04-03 22:05:56.438488 2019-04-03 22:05:56.438488 15941
|
||||||
|
fdroid 2019-04-04 19:10:50.190317 2019-04-04 19:10:50.190317 15947
|
||||||
|
xmpp 2019-04-07 08:06:59.429965 2019-04-07 08:06:59.429965 15988
|
||||||
|
blogging 2019-04-09 12:13:20.852976 2019-04-09 12:13:20.852976 16006
|
||||||
|
socialmedia 2019-04-14 19:07:37.081635 2019-04-14 19:07:37.081635 16034
|
||||||
|
c64 2019-04-14 19:56:40.692197 2019-04-14 19:56:40.692197 16036
|
||||||
|
|
||||||
|
For every tag, i removed the line with the latest date, and noted their IDs, then went on to the
|
||||||
|
``statuses_tags`` table, which looked like this (i don’t put all rows here as it would take a lot
|
||||||
|
of space):
|
||||||
|
|
||||||
|
.. code-block:: sql
|
||||||
|
|
||||||
|
COPY public.statuses_tags (status_id, tag_id) FROM stdin;
|
||||||
|
101862091116861098 15929
|
||||||
|
|
||||||
|
Here, i had to look for the IDs noted before in the last column (ie. at the end of a line), and
|
||||||
|
for every line i change the newer ID to the old ones (15929 became 237, and so on).
|
||||||
|
|
||||||
|
Finally, the ``account_tag_stats`` table:
|
||||||
|
|
||||||
|
.. code-block:: sql
|
||||||
|
|
||||||
|
COPY public.account_tag_stats (id, tag_id, accounts_count, hidden, created_at, updated_at) FROM stdin;
|
||||||
|
1 15929 0 f 2019-04-03 11:49:21.810564 2019-04-03 11:49:21.810564
|
||||||
|
|
||||||
|
Here, the ID i was looking for is in the second column, so i used the magical ViM regex
|
||||||
|
``^[0-9]\+\t\(15929\|15941\)`` (but with all the other IDs in the parentheses, too), and again
|
||||||
|
changed the new IDs to the old ones.
|
||||||
|
|
||||||
|
I still can’t believe i didn’t mess up the database too much, but it loaded just fine, and the
|
||||||
|
``db:migrate`` command executet just fine.
|
||||||
|
|
||||||
|
I also had some problems with so nginx reverse proxying, but it all turned out well, and my
|
||||||
|
instance is back online after a year. Come follow me at `@gergely@polonkai.eu
|
||||||
|
<https://social.polonkai.eu/@gergely>`_!
|
||||||
Loading…
Reference in New Issue
Block a user