Database Diagram for RootsMagic version 9 – there are probably more table relationships to be added.
Exploit your RootsMagic family tree database with SQLite Tools
Database Diagram for RootsMagic version 9 – there are probably more table relationships to be added.
RootsMagic 9.0.0.0 was released on Feb 27, 2023 with some new features necessitating changes to the database design. All previous version #database files undergo an upgrade procedure to the RM9 structure and are not backward compatible. However, #RM9 and #RM8 table structures are identical with the following exceptions:
9 new fields named color1, …, color9 have been added in support of the ColorCode sets 1 through 10. The original “color” field is set 1.
New tables added in support of the new Associations feature. They bear a resemblance, respectively, to the EventTable and FactTypeTable.
New table added in support of Saved Criteria Search and Saved Criteria Group. So far:
New OwnerType values have been added in support of the new tables. These will be identified as encountered through usage and testing.
Table | RecType | OwnerType | OwnerID | |
Payload | 1 (SavedCriteriaSearch) | 8 | 0 | |
Payload | 2 (SavedCriteriaGroup) | 20 | TagTable.TagValue, GroupTable.GroupID | |
FANTable | 19 | CitationLinkTable, MediaLinkTable,… |
This is a tab-delimited result of a query of the tables.
This script was prompted by a discussion in the RootsMagic Community on Feb 10 ’23: Notes are duplicated. The duplicate begins with MERGED NOTE. It is not uncommon, viz this Google search for the MERGED NOTE flag generated by RootsMagic. Its results return many pages, from sites.rootsmagic.com and from other sites with content from RM databases. There must be many more unpublished cases among users who have used RM’s Merge functions.
From RM4 on, merging of persons preserved the Person (General) Note value of the secondary person by concatenating it with that of the primary person separated by the string ‘– MERGED NOTE ————‘ and some whitespace. In some cases, both persons had the same Note but RM concatenated them anyway. And in others, there have been multiple merges into a person with duplication of notes occurring in any pattern.
Complicating the test for duplication is that some notes are identical except for the loss of trailing whitespace (when GEDCOM is involved) or the dropping of a symbol such as “<” or perhaps some punctuation.
This script parses the PersonTable.Note into the parts between the MERGED NOTE flags, keeps the alphanumerically distinctly different parts and concatenates them into a new Note with a symbolic demarcation ‘-+-+-+-+-+-+-+-‘ that is less brazen but still useful. It presents a table with both the NewNote and the OldNote for inspection.
Updating the PersonTable with the new Note is done by executing a selected statement within a comments block. The original flagged Notes are copied into a temporary table from which they can be restored, prior to re-executing the script after updating PersonTable with the revised Note.
Uses Common Table Expression (CTE) and the REGEXP_REPLACE() function and was tested on SQLiteSpy 1.9.15 Win32. Any current SQLite manager that has been compiled with a relatively recent SQLite source code (CTE was introduced a few years ago) and has an extension or includes the REGEXP_REPLACE() function using the same syntax as SQLiteSpy should work.
AND PersonID <= last_PersonID_checked;
2023-02-17 Script revised for easier comparison of new vs old Notes and name changed.
With all the excitement surrounding how well Artificial Intelligence is doing to write papers that can fool teachers and having seen someone post a message about using it to generate some software coding, I was curious to see what the ChatGPT interactive AI research tool could say about creating a SQLite query of a RootsMagic database. The answer, at the moment, is ‘not much’ but it does tell you how to get started with the SQLite Command-Line Interface (CLI). The composition itself is very good!
You can use the SQLite command-line interface to query your RootsMagic database. To do this, follow these steps:
sqlite3 database_file
SELECT name FROM sqlite_master WHERE type='table';
.schema people
SELECT * FROM people;
This will display all rows and columns in the “people” table. You can use various clauses in your SELECT statement, such as WHERE, GROUP BY, and ORDER BY, to filter and sort the results.
I hope this helps! Let me know if you have any questions. (How polite and generous is that!?)
RootsMagic 7 uses the Firebird (Oops! got that wrong- that’s pre-RM4) open-source database engine to store and manage data in its databases. Firebird is a powerful and reliable database engine that is well-suited for use in genealogy software like RootsMagic. It is capable of handling large amounts of data and can support a wide range of data types and structures. Firebird is also designed to be easy to use and maintain, making it a good choice for software like RootsMagic that is intended for use by non-technical users.
Firebird is based on the InterBase database engine, which was developed by Borland in the 1980s. Firebird was created as an open-source fork of InterBase in the early 2000s, and has since become a popular choice for a variety of applications that require a robust and reliable database engine.
This procedure responds to a call for help from a RootsMagic user who had received a 40,000 person GEDCOM file in which all the surnames were in capital letters and which he wanted to integrate with a database in which surnames are mixed case. How could the SURNAMES be converted to Surnames without editing in RM each of the 40k people or a Search and Replace for each unique name (numbering in the thousands)? I suggested a few ways (see Alternative Solutions below) but realised that a wholly SQLite solution working directly on the database might be do-able and be the best.
2022-11-02: I have to admit that I undertook the challenge of doing this procedure without realising that NameClean had the ALL-CAPS conversion feature. I may have had no prior need to use it and merrily presumed that the requestor had hit a dead end trying to do it in RM7. While NameClean executes the updates much more slowly than mine does, that may be of no serious consequence for a one-time operation. And, to my surprise, NameClean does have some inventory of Scottish “Mac…” surnames which it defaults to, rather than upper-casing the following letter, regardless, i.e., MACRON (French) does not become MacRon but Macron, MACKINTOSH becomes Mackintosh whereas MACNEILL=>MacNeil, etc. “MC” is likely rule-based: MCCOY=>McCoy.
The current release of SQLite 3 does not have a Titlecase or Propercase conversion function but at least one SQLite manager, SQLiteSpeed, has an internal extension with it. I tried it first with mixed results on a small set of test names (second column in the table below). The white cells have acceptable results, yellow unacceptable, peach ambiguous because the result might be acceptable for one person but not for another (cultural variation…).
Scouring the Internet, I turned up a very clever recursive query for Titlecase conversion that could work in any current-enough SQLite manager as it uses the standard functions. Tried it with SQLiteSpeed, SQLiteSpy and SQLite Expert Personal. The first two seem to have inconsistent support for Unicode, e.g., in the lower() or substr() functions while Expert provided the best results and execution was very fast.
Considering the procedure, I thought the conversion should only apply to names that are ALL CAPS and not to any others so as not to undo correct entries such as the green test names. That prompted the REGEXP filter which behaved differently on Spy and Expert and not at all on Speed but I eventually settled on one using standard functions common to all.
Further, there should be an intermediate step to inspect the converted names before applying them to the database NameTable to allow for deletions and corrections of undesired results that can only be found through inspection.
The end result is a SQL file that is not a single script intended to be executed in one go. Rather, it contains a series of short scripts enclosed as comments, each being a step to be executed in order. If you execute the whole file accidentally, there is no result because everything is commented. Each step executes very quickly but for the user interaction. The slowest execution is the actual update of the NameTable which still took only 35s for 35,000 records on a near-obsolete i5 Windows 10 laptop and hard-disk drive.
29 Oct 2022
A user with this problem could not open their RM7 database file, contacted Tech Support which could not help and was referred by the agent to me. I warned them that there was probably nothing I could do because any version of SQLite I had would report the same error as RM7’s SQLite database engine and be unable to open it. Sure enough, here’s what SQLiteSpy reported:
As I could not open the .rmgc file with SQLiteSpy, I had a look at it with a text editor (PSPad in this case; NotePad++ is a good alternative), thinking that maybe the header got corrupted. Here’s what I saw:
Two things (highlighted) jumped out at me: “PK” and a filename ending in “.rmgc”. Was this actually a RootsMagic 7 backup file whose extension had been changed from “.rmgb” to “.rmgc”? The RM backup file, probably even from the era of Family Origins, is a data compressed ZIP file and I had used, as early as the ’80’s, the original PKZIP utility for MS-DOS – there’s that “PK”. Sure enough, changing the extension to “.zip” allowed Windows File Manager, which integrates the Zip functions, to look inside and see that there was a single file in the archive whose uncompressed size was 3 times the size of the compressed backup file. Its name was actually “T.Thornley.rmgc”, a pre-RM8 RM database filename.
Then, having changed the extension from “.zip” to “.rmgb”, RM7 happily extracted “T.Thornley.rmgc” from the Zip/Backup file using File>Restore and successfully opened it. Here’s what the header looks like for the resulting “.rmgc” file, or any unencrypted SQLite 3 file, including any RM8 “.rmtree” database file:
Apparently, the user had been changing backup filenames to the “.rmgc” extension for some time as they reported that they could not “open” a backup dated later than 2020. I can see how this might be possible due to a misunderstanding of the RM Backup/Restore procedures and a trap in the RM User Interface: the backup dialog allows the user to change the extension when naming the backup file. It probably should append “.rmgb” (or “.rmbackup” for RM8) regardless; after all, that extension is required for RM to find its backups, distinct from “.zip” files and in the clutter of other file types.
If one uses File > Open to the mislabelled “.rmgc” file, there is no immediate error as in the case of RM7 but a series of dialogs having to do with conversion to RM8 until the Import starts and hangs with this “Send error report” dialog:
One would infer that it was an unencrypted database file missing a required table but that is not the case. The correct error message is provided by RM7 and it does not freeze up as is the case with RM8.
Just learned of some interesting things about SQLite in this article that popped up in my Google News feed: In Praise of SQLite by Nikos Vaggalis, dated 28 July 2022. The latest version of SQLite is 3.39.2. I don’t monitor releases so this story made me aware of some significant changes delivered in 3.39.0 and other developments in the library and in extensions. I’ll highlight some that I think will be of potential interest to RootsMagic Toolmakers but recommend you read the original report. Check your current SQLite manager for updates that may bring it up to using the 3.39.x library.
TestExternalFiles by Richard J. Otter is a new Windows utility that helps identify these issues:
That #media is one of the largest tags in the cloud in the sidebar says something about needed improvements in RootsMagic’s media management over its lifetime to date. Previously published scripts or apps may also have addressed one or more of these issues but what is novel or different about TestExternalFiles is:
TestExternalFiles has no user interface. One edits a .ini file to specify paths to the database file under test, the root folder for the media files and the output file along with setting which options are to be run and listing the exclusions. Double-clicking on the .py or .exe in the File Manager (or otherwise launching the utility) results in a brief flash of a blank window and it’s done. It is fast! Look for and open the output file to see the results of its analysis and act on as wanted.
Download the latest release from Richard’s github server ricko2001/Genealogy-scripts where you will also find fuller documentation and a bunch of other tools he has developed for his own use. The quality of his coding reflects his professional career in software development.
As of writing, still no Book Publisher feature in RM8 and along with other persistent issues, some users wish to return to the RM7.5 database from which they had ‘upgraded’. However, the only ways back using the RM user interface are lossy:
A more complete conversion is wanted.
2023-05-04: Update re #RM9: despite the inclusion of the Book Publisher feature missing from RM8, there are RM9 users who seek a way to return to RM7 for other reasons. A tweak to a RM9 database to change its version number to RM8 opens up the solution below to carry through to RM7. See Database Revert RM9 to RM8 to RM7.
Both RM7 and RM8 use similar SQLite databases so, obviously, a transfer of data through SQLite operations is the most direct means possible. However, “similar” is not “identical” and there are challenges with transferring Tasks, TreeShare, FamilySearch, Custom Reports and Report Settings and other File Settings, History, Bookmarks, et al. The script above provides the most thorough and complete conversion achieved to date and is built on and supersedes:
For a full understanding of what has gone into the current procedure, they remain useful references.
As for earlier versions, the order in which Citations for a given fact are presented in RM8 is the order in which they were added. Some users wish for them to be sorted alphabetically by name.
A brute force method was described for earlier versions at Source Citation Sort . In that case, the name of the Master Source was presented in Citation Manager so the CitationTable itself had to be reordered according to the Source Name and required links from tables for media and web tags to track the changes. With the introduction of TreeShare, there was a further complication for links between Citations and Ancestry.com. Revisions to the database structure in RM8 cause that script to fail.
Luckily, those changes in database design for RM8 that enabled reusable citations (master citation concept) make the job of sorting the citations easier and faster than for pre- RM8. It is the new CitationLinkTable that must be re-ordered and that has no knock-on effect for media, web tags or Ancestry.com links which remain linked to the CitationTable which does not need to be re-ordered. The new CitationName field requires that it be the second sorting field after the Source Name.
To sort citations in a RM8 database, load the following script into a SQLite manager with a fake #RMNOCASE extension and execute it against the target .rmtree file (backup first!). Note that the database file likely must not be open in RootsMagic as it may cause a ‘file is locked’ error on the outboard SQLite. When you reopen the database in RM8, run Files > Tools > Rebuild Indexes to clear out indexing errors from the outboard process reported by RM8 Test Integrity.
This post is the outcome of the Forum discussion CitationSort.sql giving an error when run..,