Exploit your RootsMagic family tree database with SQLite Tools
https://github.com/ricko2001/Genealogy-scripts
Richard Otter is a skilled, professional developer who is also a RootsMagic user and has been building a library of scripts that are worthwhile checking out for possible solutions for your own needs. He publishes them on GitGub, list of them screenshotted below 2023-05-19. Check for updates and fuller descriptions. He has posted notices of significant ones in the Forum and, hopefully, will continue to do so.
This project arose from a request Fix & Merge Hundreds of Newspapers.com Sources in the Forum. The poster is a heavy consumer of the Newspapers.com sources through RM’s TreeShare with Ancestry.com and had issues with:
Because the sources were imported via TreeShare, they are Ancestry Record type, i.e., they are created using the built-in Ancestry Record Source Template. Built-in Source Templates are uneditable through the RM user interface but are defined in the same table that holds user-defined templates. Thus, the built-in templates can be modified by using SQLite to edit entries in the SourceTemplateTable. We can address Issues #3 and #4 by modifying the Footnote and Bibliography sentence templates in the Ancestry Record template. That will be of benefit also to citations having empty values from some other Ancestry Collections (see Ancestry TreeShare – Impact).
Issues #1 and #2 are more challenging because the values of the source and citation variables that appear in the Footnote and Bibliography sentences are stored in a XML data structure. To solve #1, we want to “lump” all citations of a given newspaper Title under one Master Source. That requires that the data that differentiates the Master Sources for a common newspaper must be deleted or transferred from the Master Source to the Citation Details. For example, the Page # must be extracted from the Source Name in the SourceTable and moved to the Detail ([Page] variable in XML) for each Citation of that Source in the CitationTable. There are more steps than that alone for each of that one newspaper’s multiple Master Sources and Citations.
Once all the data manipulations are complete, there will be multiple identical Master Sources for a given newspaper Title. RM’s AutoMerge Sources function can finish the job.
The database undergoing modification was from RM7, hence the screenshots are of RM7. However, the solution also works with RM8 and RM9.
Conversion from #RM9 to #RM8 is pretty trivial with little loss except for RM9’s new features. From RM8 to #RM7 is much more complex given the significant structural changes between the two. To convert from RM9 to RM7 is simply a two-stage process with a RM8 compatible file as an intermediary. It’s possible to combine the two into one procedure but that’s for another time.
That there should be a variance in Tasks is unsurprising because of the structural changes between RM7 and RM9; Task Folders from RM9 could be non-existent in RM7.
The tiny variance in Citations is perhaps more surprising as it might be expected to go in the opposite direction, e.g., citations for Associations in the original RM9 database would be absent in RM7 and unrecoverable in the re-upgrade to RM9. In this case, however, it is because a few of the original RM9 Citations were reused and there were 6131 Citation Links which were transformed to 6131 individual Citations in the downgrade from RM8 to RM7.
The loss of Associations in the downgrade has no effect on the Events count as they are records in a different table, only reported in RM9’s Enhanced Properties List which also reveals the loss of Task folders.
Of course, the database properties lists do not tell the whole story. It would also be unsurprising to find that Notes, especially formatting, may not be preserved in a downgrade or upgrade due to the major reiterations of the Note Editor code. Certainly, new features in RM8 or RM9 will not be supported in a lower version and those that are database dependent, including Associations, Color Code Sets and Labels, and Saved Search Criteria cannot survive a round trip from a higher version to lower and back.
The only barrier to RM8 opening a RM9 .rmtree file is the version number of the database. For RM8, it cannot deal with a version higher than 8000; RM9’s is 9000 while RM7’s is 6000 (as it is for RM6). This statement opens the door to RM8:
UPDATE ConfigTable
SET DataRec=
CAST(
REPLACE(DataRec,'<Root><Version>9000</Version>',
'<Root><Version>8000</Version>')
AS BLOB
)
WHERE RecID=1
;
The next step is only needed to prevent RM8 from offering the Association fact type (which it cannot support) as a conventional individual type which one could use to add such events, leading to confusion should the database be reopened in RM9.
DELETE FROM FactTypeTable WHERE FactTypeID=902; -- Association Fact Type --DELETE FROM EventTable WHERE EventType=902; -- Association individual event, if any
The second statement is only needed should such Association individual events have been created before upgrading back to RM9.
A possible enhancement to this down-conversion from RM9 to RM8 might be to convert Associations in RM9 to either Shared Events or simple individual type events with the name of the other person in the event description.
The heavy work in developing a procedure for this down-conversion was covered in Convert RM8 Database to RM7.
In response to a request in the Forum, Adding Surname to Truncated Source Names in RM9, posted here is a script that may be of use to others. The user’s objective is to facilitate distinguishing which Master Sources (among many similar source names derived from imports from FamilySearch, Ancestry, et al) are relevant to families or persons of interest without clicking down through Citations, Citation Uses and dead ends from which he needs to back out to go onto the next. With 32k Sources, 32k Citations (he’s a ‘source-splitter’) and 113k ‘uses’ among 7k people and 3k families, one can appreciate the scope of his challenge.
This batch script appends to the Source Name the unique surnames of all persons in the database who have a citation or ‘use’ of a Master Source in their profile. The list of surnames is enclosed in drawing symbols: ╣surnamelist╠ acting as bookends. The resulting extended name is gracefully truncated at 256 characters if caused by the operation, the maximum RM9 supports in a drag’n’drop transfer; the value is easily changed in four places in the same statement.
The script execution creates a series of temporary Views (in-memory queries) to build
the final View “TruncNewName” from which the SourceTable is updated with the
╣surnamelist╠ appended. These Views are lost when the SQLite manager closes the database.
At the start of the script after REINDEXing against the fake RMNOCASE collation, the SourceTable is updated with Names stripped of previous ╣surnamelist╠. Therefore, the script can be rerun again after changes have been made in the database.
Because the database has been REINDEXed against the fake RMNOCASE, the RM Rebuild Indexes tool must be run on returning to work on the database with the RootsMagic app.
Execution time for the script on the sample database with 32k sources is 20-25 seconds on a 5-yr old i5, middling laptop with HDD so it is not prohibitively long to use repeatedly. I do not know how the time would scale with larger databases.
Requires a SQLite manager that has a “fake RMNOCASE collation” and supports REGEXP_REPLACE(). Script was developed and tested with SQLiteSpy 1.9.16 64-bit and the fake RMNOCASE extension from
RMNOCASE – faking it in SQLiteSpy. Backup before using!
For source ‘lumpers’, the string of surnames might well be too long for the Source Name. Even for ‘splitters’, some source such as a census page for a residential school, hospital, prison could have a long list of surnames. The current Sources view in RM9 is amenable to showing only 100-150 characters depending on screen size and the position of the divider while a drag’n’drop truncates at 256. That may call for two things:
Working on this project has got me thinking about a means to differentiate what RM thinks are duplicate citations when they differ in Media or Web tags. Stay tuned…
And please comment here or in the Forum on whether this script does anything positive or negative for you!
They say a picture is worth a thousand words. This page compares pictures of the #RM9 and Heritage 2023 databases (both SQLite 3) as automatically created by the same tool.
An Entity-relationship Diagram (ERD) says a lot about a database. @patjones recently posted a RM9 Database Diagram with many of the linkages or relationships among its tables that are made on-the-fly by the SQL queries from the RM software or the ones found on this site. What a database Entity Relationship Diagram editor generates from a database’s self-definition includes the tables, fields and types, stored Triggers (actions on 1 or more tables by some database event) and stored Views (stored queries). So it is interesting to compare the automatically generated ERDs of the RM9 database against that of the Heredis 2023, from the same editor, DBeaver.
| Object | RootsMagic 9.0 | Heredis 2023 |
| Tables | 29 | 65 |
| Views | 0 | 0 |
| Indexes | 46 | 123 |
| Table Triggers | 0 | 156 |
The comparison table and diagram show that the Heredis database is considerably more complex than RootsMagic 9 with more than double the number of tables and nearly triple the number of indexes which accelerate the association of rows across tables and speed up searches. And more divergent is the extensive use of Triggers by H and their complete absence in RM. What do these numbers say about the two databases?
That there are more tables in H than RM is from a couple of causes:
Neither has stored views (stored queries) so that tells us nothing. I’ve used temporary Views on RM databases, lasting until the database is closed by the sqlite manager. It’s possible that both applications do, too, but we cannot see them from outside the app’s sqlite engine.
With double the number of tables, it stands to reason that there could be double the number of indexes. I don’t know why there should be triple but, in the past, I’ve suggested an index be added or revised in RM. SQLite is itself very good at optimising queries and may spawn an index that it thinks is appropriate unless there is a pre-defined one that includes all the key fields of the query. Maybe H developers have attempted to cover all the possible bases with explicit indexes.
What I have yet to find out is if there is any custom collation sequence in H as there is in RM which has given us database explorers so much grief.
This is the most pronounced difference between RM and H. H has many routine operations embedded in the database, being executed by the database engine, rather than in the higher level code which then has to talk to the database engine to make the same thing happen. For example, H has a trigger named “MajDateMediasEvenementInsert” which automatically updates the ‘Date modified’ value in its ‘EventTable’ when a link has been added to their ‘MediaLinkTable’. Looks like this:
CREATE TRIGGER "MajDateMediasEvenementInsert"
AFTER INSERT ON LiensMediaEvenement
FOR EACH ROW
BEGIN
UPDATE Evenements SET DateModification=MAX(NEW.DateModification, Evenements.DateModification) WHERE Evenements.CodeID=NEW.XrefEvenement;
END;Maybe not the best example for comparison because I don’t think RM would update the EventTable when a media item is tagged to an event.
Having searched around for a (free) application that could create an Entity Relationship Diagram (ERD) from a SQLite 3 database, I landed on one that might be described as an Integrated Development Environment (IDE) for databases, the DBeaver Community Edition. It is self-described as a “Universal Database Tool” and I’ve only superficially scratched its surface figuring out how to connect a SQLite database, generate the ERD from its definition and edit it to illustrate relationships. But it is much more powerful than that as it looks to be a multi-database manager with which one can query the database data. And there is a Pro version which adds many more capabilities.
Database Diagram for RootsMagic version 9 – there are probably more table relationships to be added.
RootsMagic 9.0.0.0 was released on Feb 27, 2023 with some new features necessitating changes to the database design. All previous version #database files undergo an upgrade procedure to the RM9 structure and are not backward compatible. However, #RM9 and #RM8 table structures are identical with the following exceptions:
9 new fields named color1, …, color9 have been added in support of the ColorCode sets 1 through 10. The original “color” field is set 1.
New tables added in support of the new Associations feature. They bear a resemblance, respectively, to the EventTable and FactTypeTable.
New table added in support of Saved Criteria Search and Saved Criteria Group. So far:
New OwnerType values have been added in support of the new tables. These will be identified as encountered through usage and testing.
| Table | RecType | OwnerType | OwnerID | |
| Payload | 1 (SavedCriteriaSearch) | 8 | 0 | |
| Payload | 2 (SavedCriteriaGroup) | 20 | TagTable.TagValue, GroupTable.GroupID | |
| FANTable | 19 | CitationLinkTable, MediaLinkTable,… |
This is a tab-delimited result of a query of the tables.
This script was prompted by a discussion in the RootsMagic Community on Feb 10 ’23: Notes are duplicated. The duplicate begins with MERGED NOTE. It is not uncommon, viz this Google search for the MERGED NOTE flag generated by RootsMagic. Its results return many pages, from sites.rootsmagic.com and from other sites with content from RM databases. There must be many more unpublished cases among users who have used RM’s Merge functions.
From RM4 on, merging of persons preserved the Person (General) Note value of the secondary person by concatenating it with that of the primary person separated by the string ‘– MERGED NOTE ————‘ and some whitespace. In some cases, both persons had the same Note but RM concatenated them anyway. And in others, there have been multiple merges into a person with duplication of notes occurring in any pattern.
Complicating the test for duplication is that some notes are identical except for the loss of trailing whitespace (when GEDCOM is involved) or the dropping of a symbol such as “<” or perhaps some punctuation.
This script parses the PersonTable.Note into the parts between the MERGED NOTE flags, keeps the alphanumerically distinctly different parts and concatenates them into a new Note with a symbolic demarcation ‘-+-+-+-+-+-+-+-‘ that is less brazen but still useful. It presents a table with both the NewNote and the OldNote for inspection.
Updating the PersonTable with the new Note is done by executing a selected statement within a comments block. The original flagged Notes are copied into a temporary table from which they can be restored, prior to re-executing the script after updating PersonTable with the revised Note.
Uses Common Table Expression (CTE) and the REGEXP_REPLACE() function and was tested on SQLiteSpy 1.9.15 Win32. Any current SQLite manager that has been compiled with a relatively recent SQLite source code (CTE was introduced a few years ago) and has an extension or includes the REGEXP_REPLACE() function using the same syntax as SQLiteSpy should work.
AND PersonID <= last_PersonID_checked;
2023-02-17 Script revised for easier comparison of new vs old Notes and name changed.
With all the excitement surrounding how well Artificial Intelligence is doing to write papers that can fool teachers and having seen someone post a message about using it to generate some software coding, I was curious to see what the ChatGPT interactive AI research tool could say about creating a SQLite query of a RootsMagic database. The answer, at the moment, is ‘not much’ but it does tell you how to get started with the SQLite Command-Line Interface (CLI). The composition itself is very good!
You can use the SQLite command-line interface to query your RootsMagic database. To do this, follow these steps:
sqlite3 database_file
SELECT name FROM sqlite_master WHERE type='table';
.schema people
SELECT * FROM people;
This will display all rows and columns in the “people” table. You can use various clauses in your SELECT statement, such as WHERE, GROUP BY, and ORDER BY, to filter and sort the results.
I hope this helps! Let me know if you have any questions. (How polite and generous is that!?)
RootsMagic 7 uses the Firebird (Oops! got that wrong- that’s pre-RM4) open-source database engine to store and manage data in its databases. Firebird is a powerful and reliable database engine that is well-suited for use in genealogy software like RootsMagic. It is capable of handling large amounts of data and can support a wide range of data types and structures. Firebird is also designed to be easy to use and maintain, making it a good choice for software like RootsMagic that is intended for use by non-technical users.
Firebird is based on the InterBase database engine, which was developed by Borland in the 1980s. Firebird was created as an open-source fork of InterBase in the early 2000s, and has since become a popular choice for a variety of applications that require a robust and reliable database engine.
This procedure responds to a call for help from a RootsMagic user who had received a 40,000 person GEDCOM file in which all the surnames were in capital letters and which he wanted to integrate with a database in which surnames are mixed case. How could the SURNAMES be converted to Surnames without editing in RM each of the 40k people or a Search and Replace for each unique name (numbering in the thousands)? I suggested a few ways (see Alternative Solutions below) but realised that a wholly SQLite solution working directly on the database might be do-able and be the best.
2022-11-02: I have to admit that I undertook the challenge of doing this procedure without realising that NameClean had the ALL-CAPS conversion feature. I may have had no prior need to use it and merrily presumed that the requestor had hit a dead end trying to do it in RM7. While NameClean executes the updates much more slowly than mine does, that may be of no serious consequence for a one-time operation. And, to my surprise, NameClean does have some inventory of Scottish “Mac…” surnames which it defaults to, rather than upper-casing the following letter, regardless, i.e., MACRON (French) does not become MacRon but Macron, MACKINTOSH becomes Mackintosh whereas MACNEILL=>MacNeil, etc. “MC” is likely rule-based: MCCOY=>McCoy.
The current release of SQLite 3 does not have a Titlecase or Propercase conversion function but at least one SQLite manager, SQLiteSpeed, has an internal extension with it. I tried it first with mixed results on a small set of test names (second column in the table below). The white cells have acceptable results, yellow unacceptable, peach ambiguous because the result might be acceptable for one person but not for another (cultural variation…).
Scouring the Internet, I turned up a very clever recursive query for Titlecase conversion that could work in any current-enough SQLite manager as it uses the standard functions. Tried it with SQLiteSpeed, SQLiteSpy and SQLite Expert Personal. The first two seem to have inconsistent support for Unicode, e.g., in the lower() or substr() functions while Expert provided the best results and execution was very fast.
Considering the procedure, I thought the conversion should only apply to names that are ALL CAPS and not to any others so as not to undo correct entries such as the green test names. That prompted the REGEXP filter which behaved differently on Spy and Expert and not at all on Speed but I eventually settled on one using standard functions common to all.
Further, there should be an intermediate step to inspect the converted names before applying them to the database NameTable to allow for deletions and corrections of undesired results that can only be found through inspection.
The end result is a SQL file that is not a single script intended to be executed in one go. Rather, it contains a series of short scripts enclosed as comments, each being a step to be executed in order. If you execute the whole file accidentally, there is no result because everything is commented. Each step executes very quickly but for the user interaction. The slowest execution is the actual update of the NameTable which still took only 35s for 35,000 records on a near-obsolete i5 Windows 10 laptop and hard-disk drive.
29 Oct 2022
You will be repeatedly prompted in subsequent steps.