Source Citation Sort #sources #citations

A longstanding wish is for RootsMagic to list sources alphabetically in the Citation Manager rather than in the order they were cited for the particular fact. This would seem to be a fairly trivial programming change but as of RootsMagic 7, it has yet to be delivered. Here is a script that accomplishes the same ends by brute force – a complete reordering of the records in the CitationTable. Of course, that requires revising all other tables that point to it.
CitationSort.png
This first script is not essential to getting the Citation Manager sorted but does make inspection of the data tables a little easier. It reorders the rows in the SourceTable (the Master Sources) alphabetically by name. It requires a fake RMNOCASE collation extension and you should run RootsMagic File>Database tools>Rebuild Indexes on the database when you return to work on it. SourcesSort.sql

This script does the job for Citation Manager. It, too, requires the SQLite manager to have a fake RMNOCASE collation but should not cause a failure in the RootsMagic Integrity Check nor require its Rebuild Indexes. CitationSort.sql

Both were developed using SQLiteSpy.

I have not tested extensively so be sure to make a backup or work on a copy of your database and check that there are no unwanted consequences before committing to it as your ongoing file.

N.B. CitationSort.sql needs revision so that it will not disconnect or cross-connect Ancestry Sources linked by TreeShare in RootsMagic 7.5+. It can be used on databases that are not connected by TreeShare. There may be a similar issue for sources linked with FamilySearch through RootsMagic’s FamilySearch Central.

COPY FamilySearchID to REFN fact #refno #FamilySearch

Update 2023-11-22: #RM9 version

Problem Description

This script responds to a user’s problem posted on FaceBook: she had two databases with overlapping people in them. One was her master with people matched to FamilySearch; the other was developed independently using FamilySearch. Because the duplicated people were created independently, she cannot rely on the RootsMagic File>Compare Files tool to unambiguously pair them based on a common UID (RootsMagic’s hidden Universal Identification) because their UID’s are different. But they have been matched to the same FamilySearch persons and thus have the FamilySearch ID (FSID) in common. Unfortunately, Compare Files does not a high match make for duplicate FSIDs. That seems to be a shortcoming that should be addressed.

Duplicate Search Merge has two options:

  1. Find people with the same Ancestral File numbers (ignore all other information)”
  2. “Find people with the same reference numbers (ignore all other information)”

Wouldn’t it be nice if there was another option:

  • “Find people with the same FamilySearch ID (ignore all other information)”

Absent that third option, the user figured that if she could get the FSID into a Reference Number (Ref#) fact for each person, the databases could be combined and DSM with option 2 would reliably pair up the duplicate people. This should also be useful when people matched to FamilySearch get duplicated in a database through other avenues such as overlapping downloads from FamilySearch into the same database.

Getting the FSID into a Ref# fact is a laborious task if there are more than a few people to do. Moreover, it is complicated if the people have pre-existing Ref# facts for other purposes that must be preserved. Duplicate Search Merge compares only the first Ref# fact for a person to the first Ref# fact for each other person. Thus it requires this FSID Ref# fact to be the lowest record number of all the Ref# facts for a person in the EventTable.

Solution

Enter a SQLite script that addresses these issues enabling DSM to pair up matching FSID Ref# facts.

REFN_CopyFSID.sql.PNG
Sample of database after script has run to copy the FamilySearch ID into the first Ref# fact for each person for those persons having been matched to FamilySearch.

Download Script

REFN_CopyFSID.sql for RM versions before RM8

REFN_CopyFSID-RM9 for RM versions after RM7 (tested on a RM9 database) 2023-11-22

Description

Creates a Ref# fact containing the FamilySearch ID for each person in the database matched to a person on FamilySearch Family Tree in the format “fsid: XXXX-XXX”. This enables the Duplicate Search Merge (DSM) option “Find people with the same reference numbers (and ignore everything else)” to pair up people with the same FSID Ref#.

This script shuffles existing Ref# facts to follow the FSID Ref# facts it creates by copying the former out to a temp fable xRefnBak, deleting the originals from the EventTable and then appending them after the FSID Ref# facts are created.

The script also preserves and pushes to the front existing FSID Ref# facts for a person not currently matched in the database to Family Search, i.e., the FSID has been somehow lost or the fact was manually added. This feature is of uncertain value.

A temporary table xRefnBak is created by the script and is deleted when the SQLite manager closes the database.

Requirements & Caveats

Requires the REGEXP extension which is not included in all SQLite managers; SQLiteSpy does support it.

N.B.: the script assumes there are no sources, media, or any other element attached to the existing Ref# facts and makes no attempt to preserve those linkages nor to delete the records in the corresponding tables. It is possible that some will become erroneously attached to new FSID Ref# facts.

Discussions & comments from Wikispaces site


gualco

REGEXP error

gualco
05 September 2016 23:03:56

Thank you so much for this script!!!

I tried using it and got an error. I’m wondering if I am just using it incorrectly–does it need to be run in pieces? do I need a different SQL app? etc.

When I try to run this script, I get this error:

no such function: REGEXP.

This is what I did:
1. Open SQLite Expert Personal 4.2.0.693 (x86)
2. Open my RootsMagic DB file within SQLite
3. Click on the EventTable in the left sidebar
4. Click on the SQL tab and paste in the script
5. Click the “Execute SQL” button
6. Error pops up

Gina


ve3meo

ve3meo
06 September 2016 00:13:54

Hi Gina, use SQLiteSpy. It supports REGEXP.

Tom

Reports, Narrative, Jerry Bryan Trick #reports

Rev 2016-08-16
This is an extension of DummyFamily-Add.sql described below with added constraints so that the trick is applied only to those childless, spouseless persons having events other than Birth or Death or more than two of those or BD events with notes or having a General Note or an Alternate Name. Requested by wiki member aefgen in the hope that it would reduce the number of instances of “excessive white space”, a collateral consequence of the trick.
DummyFamily-Add2.sql rev 2016-08-20 added having a General Note or an Alternate Name


These two queries carry out the trick first reported by Jerry Bryan in the RootsMagic Forums in the topic Register Style Reports, Individuals Without Children or Spouses.

DummyFamily-Add.sql

/* DummyFamily-Add.sql
Based on trick developed by Jerry Bryan to cause Narrative Descendant reports
to output childless, spouseless children in their own generation, along with their
married or parental siblings.
 
Adds dummy family(ies) for this(all such) child(ren), based on the runtime parameter,
to the FamilyTable. Requires SQLite Export Personal or equiv to support selection of a single
child.
 
After completing the report, run DummyFamily-Delete.sql; it is probably less risky, however,
to run the first query and report on a copy of the database and then delete the copy.
2012-01-15 ve3meo
*/
 
INSERT INTO FamilyTable
 (FatherID)
SELECT ChildTable.ChildID FROM ChildTable
WHERE
 CASE
  WHEN @RIN NOT LIKE '' THEN ChildTable.ChildID = @RIN
  ELSE 1
 END
AND
 ChildTable.ChildID NOT IN (SELECT FatherID FROM FamilyTable)
AND
 ChildTable.ChildID NOT IN (SELECT MotherID FROM FamilyTable)
;

DummyFamily-Delete.sql

/* DummyFamily-Delete.sql
Reverses the changes made by DummyFamily-Add.sql to the FamilyTable.
 
After completing the report, run DummyFamily-Delete.sql; it is probably less risky, however,
to run the first query and report on a copy of the database and then delete the copy.
2012-01-15 ve3meo
*/
 
DELETE FROM FamilyTable
WHERE
 FatherID
 IN
  (
   SELECT FatherID FROM FamilyTable
   WHERE
    CASE
     WHEN @RIN NOT LIKE '' THEN FatherID = @RIN
     ELSE 1
    END
   AND
    MotherID IS NULL
   AND
    ChildID IS NULL
  )
;

Discussions & comments from Wikispaces site


aefgen

Re: Dummy Family Add script (“Jerry Bryan trick”)

aefgen
16 August 2016 21:29:35

Tom, thanks – on first examination, this revision definitely improves the report greatly. The choice of which individuals to carry forward to the next generation and which to keep under the parents works well and eliminates much but not all of the excess white space. I will look more carefully tomorrow and see if it might be tweaked further. One problem was that on trying to backup the file, RM said it encountered an error, and when I then used the database tool to check the file integrity, it gave me an SQ Lite error 1 – no such collation sequence: RMNOCASE. The program then froze and I had to use task manager to stop RM. But when I went back in and ran the database tools, they reported that all was well, and I was then able to backup the file. We’ll see if that happens again. – Alex


thejerrybryan

thejerrybryan
20 August 2016 00:18:32

The reports that I post process are usually 50 to 60 pages, so the scale is not quite so grand as yours. Nevertheless, I try to do as much of the cleanup as possible with global replaces. If you want to go that route, you will have to play with it to see what works for your needs and what doesn’t. I usually try to do the global replaces in a text editor, processing the RTF file produced by RM before it has been touched by Microsoft Word. That requires learning a little bit about RTF tags. The other approach (and sometimes I do both) is to do the global replaces from within Microsoft Word. You have to turn on Word’s equivalent of what WordPerfect used to call “Reveal Codes” to expose white space characters to global replace. It’s tricky business, either with a text editor or with Word, but if you can figure out what meets your needs you can save a huge amount of time with global replaces. In case you want to try the RTF file plus text editor option, I wouldn’t recommend Notepad for a file that big. I would recommend Notepad++ instead – a very powerful and free text editor.

On your other questions, I really haven’t looked at them. What you are doing seems much fancier than what I do, and I have to admit that I don’t use Tom’s script. I still do it kind of manually where I add dummy children to anybody I want to force into the next generation in a narrative report. I leave such dummy children in my production database at all times and when I’m preparing a report for a family reunion I copy my production database into a reporting database that I can manipulate as necessary to produce the report. I use File->Copy, not Drag-and_Drop or GEDCOM Export/Import to make the copy. Then, I have an SQLite script that essentially deletes the dummy children without also deleting the FamilyTable entries for the parent of the dummy children. You can’t do the delete of dummy children that I’m talking about from within RM itself because RM is smart enough to also delete the FamilyTable entry for the person to be carried into the next generation, and using the FamilyTable entries to carry the person into the next generation is the whole point of the exercise. So my delete of the dummy children is an incomplete delete which is exactly what is needed. But I’m afraid that my rather manual technique doesn’t scale up very well to your use case. That’s part of the reason that Tom wrote his script.

Jerry

Quick Start for Dummies #sql #sqlitespy

You need a SQLite manager such as SQLiteSpy to read and modify a RootsMagic database using the queries or scripts from this wiki. That’s because RootsMagic incorporates the SQLite 3 database engine to write and read its database files.

SQLite acts on the database in response to SQL statements or commands it is given. These are commonly called “queries” which is a bit misleading because a query can also modify the data. A script comprises one or more statements. Because RootsMagic does not accept user inputted SQL statements, we use an outboard SQLite manager, an application that provides an interface for the user to input SQL statements to operate on a database file.

Because SQLite 3 is a free, open-source package, there are lots of applications that can open a RootsMagic file. I recommend SQLiteSpy to start with on Windows – I don’t know Mac. See SQLite Managers for download links.

You will also need to extend your SQLite manager to substitute for the secret RootsMagic RMNOCASE collation sequence having a large multilingual alphabet which is used to sort names of people, places, etc. . Some of the queries simply will not work without it, returning the error “SQLite3 Error 1 – no such collation sequence: RMNOCASE”. See RMNOCASE – faking it… to get the substitute.

OK, get ready to rumble… First and foremost, make a backup of your RootsMagic database or a copy of it that you can play with so that if it gets messed up by your queries your master is intact. If you do not know how to do that, do not pass Go but proceed directly to Jail!

With SQLiteSpy installed and opened:

  1. File > Open.
  2. Browse to the folder where your play database is located.
  3. Start typing the first letters of the database filename or enter *.rmgc to get a list of the RootsMagic database files
  4. Select your file and it will now be registered in the left panel.
  5. Click on the + signs to expand and see the list of tables.
  6. Double-click on a table name and SQLiteSpy will automatically issue a (hidden) SQL statement or query to list all of the records in that table. Results will display to the right in the middle pane . Select any cell in the results pane and you can see its content in the bottom pane.
  7. In the upper pane, type in “SELECT Surname, Given FROM NameTable;” and then press F9 or click on menu Execute > Execute SQL. The results of this user-inputted query display the same way as for the macro query of step 6.

Congrats! You have run two queries on your database.

You can copy and paste a script from this wiki into the same SQL editor space or download a script file and use File > Open SQL to load it into the editor. Read about the experiences of other beginners at How to query RootsMagic.

I have had the same database open in both SQLiteSpy and RootsMagic with little conflict. After running a script that modifies data, you may need to refresh a RootsMagic screen to see the effect; closing and reopening the database is sure to do so. RootsMagic sometimes (wrongly) leaves a lock on the database that prevents SQLiteSpy from executing a query; closing RootsMagic is the only solution.

After modifying a database outside of RootsMagic, it is advisable that when you next go to use it in RootsMagic to run the RM Database tool “Test database integrity”. If it is OK, you are good to go. However any change made to a name is likely to cause the test to fail and require that you apply the “Rebuild indexes” tool. I have yet to see any harm come to a database by the inconsistencies between the fake and the actual RMNOCASE collations.

For more on using SQLiteSpy, this tutorial (while unrelated to RootsMagic and genealogy) does provide useful explanations:
[TUTORIAL] SQLiteSpy 101 – Civilization Fanatics’ Forums

SQLite Queries #sqlitemanagers #sql

This page has not been updated since 2012 but many scripts have been added since. The tag cloud on the home page and the search control are two good ways to find pages of interest.

RootsMagic Queries

Many of these were developed around RootsMagic 4 and should also work on RootsMagic 5 and 6 with limitations. Likewise, many were developed around RM5 and should also work on RM6, with fewer limitations. The limitations are due to the changes in database definitions. RM5 added the ResearchItemTable and changed the storage of some Media Properties from the tag to the media item. RM6 added the URLtable in support of WebTags. Therefore a few queries may be erroneous in dealing with Media, and/or ignore the ResearchLinkTable or the URLtable but most would be unaffected because their purpose does not include these tables.

Some SQLite managers can import a query file and some can also export a query file. Others may require you to open the file with a text editor and copy/paste the query into the SQLite manager’s query editor.

Some smaller standalone queries are combined in one file; other larger files may be multiple queries that run in seqence. Depending on the SQLite manager, you can select which statements you want to run by highlighting them or by placing the cursor anywhere in the one statement to be run. Each SQL statement must end with a semi-colon to demark it from the others.

Optionally, a single query can be expanded to begin with the CREATE VIEW command. When successfully executed, the query is embedded in the database as a virtual table which remains with the database file until it is ‘dropped’ (deleted). Some SQLite managers allow you to save a query as a view from the user interface, putting a GUI in front of an internal CREATE VIEW.

The following table lists the queries from most recent to oldest. Queries are categorized in a menu-like structure on the page Query Menu.

DescriptionPageQuery NameDate Added
Creates a Citation WebTag for each citation having an URL beginning the Citation Comments field with “http://”. Not restricted to Ancestry.com downloads.WebTags – from Ancestry.com and FTMWebTags-MakeFromAncestryComments2012-12-10
Creates Person WebTags from Citation WebTags to consolidate all WebTags pertaining to a person under the WebTags button on the Edit Person Screen.WebTags – ConsolidateWebTags-Consolidate2012-12-10
Deletes perfectly duplicate WebTags as will occur if WebTags-Consolidate is repeated or other possible cause.WebTags – ConsolidateWebTags-DeleteDuplicates2012-12-10
Sets the Media Type value for each media file to one of the four types: Image, File, Sound, Video, according to the file extension.Media Type ResetMediaTypeReset2012-12-04
Inserts an Alternate Name of type “Married” for female “Wife” spouses in a “Family” where no Alternate Surname matches the male “Husband’s” and a (family) Marriage fact exists.Names – Add MarriedName-Add_Married2012-11-26
Sets Birth and Death years as seen in the sidebar index and various other reports and displays to match the corresponding facts. NOT for use on RM4; requires RM5+ Rebuild Indexes after completion.Rebuild Indexes and Update Birth and Death YearsUpdateBirthDeathYears2012-11-14
For encoding to SortDates, dates with optional RM modifiers and decoding same.Dates – SortDate AlgorithmSortDateDecodeDev2012-11-03
Procedures involving Excel, text editor and SQLite queries to bring Ancestry hyperlinks, citations and images into RootsMagic that are otherwise lost on a straight GEDCOM import.Ancestry.com and RootsMagic 5MergeAncestryURLsToFTM
Shrinking Verbose Master Sources from Ancestry.com
2012-10-31
Cleans out unused records from most tables, including phantom citations. Follow by RM Rebuild Indexes and Compact.Delete PhantomsDeletePhantoms2012-10-27
Delete all people color-coded RED (or other color by editing the query). Follow with Delete Phantoms.

Delete all people belonging to a Named Group (edit the query to the group ID). Follow with Delete Phantoms.

Delete Many DeleteByColorCode

DeleteByNamedGroup

2012-10-27
Lists citations for Primary Name as imported from Ancestry.com, FTM, et al but hidden in RM.
Converts such invisible citations to visible Person citations.
Citations Invisible RevealedCitations, Invisible – List
Citations, Invisible – Convert to Personals
2012-09-10
Lists all addresses in AddressTable and reports how they are used.Addresses – How UsedAddressesHowUsed2012-08-17
An outline of procedures that have been used when People View throws error message “SQLite Error 1 – at most 64 tables in a join”.People View Error – at most 64 tables in a JoinPeople View Error2012-03-19
Lists the Include settings for all fact types.

Sets all fact types to be included in GEDCOM and drag’n’drop.

Stores a snapshot of all settings for fact types in an extra table in the database.

Lists the include settings stored in the snapshot.

Restores all settings for fact types from the snapshot.

Deletes the extra table storing the snapshot from the database.

Fact Inclusion Controls2012-03-12
An outline of procedures that have been used with some success to recover a corrupted RootsMagic database.Corrupt Database RecoveryCorrupt Database Recovery2012-03-10
Finds SSN facts that have not been set to Private.WebTrees Website – Pre-Processing Tools using SQLite in Visual Basic et alRMSanity.exe2012-02-06
Finds broken media links using an extension in Visual Basic.
Finds media items in the Media Gallery that are not tagged to anything in the database.
Finds media items that have been attached more than once to the Media Gallery
WebTrees Website – Pre-Processing Tools using SQLite in Visual Basic et alRMSanity.exe2012-02-06
Finds Place Details that have no owner Place.WebTrees Website – Pre-Processing Tools using SQLite in Visual Basic et alRMSanity.exe2012-02-06
This series of queries and related Windows commands revises a RM4 database and makes copies of the multiply-linked files so that the update to RM5 will preserve all the metadata.Media – Preserve Captions et al from RM4 to RM5RM4-MultiLinkedMediaDuplicationPreRM52012-01-31
One query lists all Place Detail rows from PlaceTable for which there is no master Place. Another deletes such rows.Place Details without a Place Place Details without a Place2012-01-30
The opposite direction is much easier. Needed for export to GEDCOM for third party apps that do not support Place Detail.Convert Place + Place Detail to PlaceConvert Place + Place Detail to Place2012-01-22
Compare EXIF, IPTC, XMP metadata stored in image files to the Caption, Description and other metadata stored in a RootsMagic 5 database for image files linked to its Media Gallery and copy between the files and the database.Media Metadata, Read, Write, Compare with PicasaRM5comparePicasa.bat
ImportPicasaDescriptions.sql
RM5copyMetadataToImageFileExifToolArgList.sql
RM5copyMetadataToImageFileExifToolArgList.bat
RM5copyMetadataToRM.sql
RM5copyMetadataToRM.bat
2012-01-17
Lists those image files that have not been opened by RM, hence nothing in the thumbnail field. With comments leading to other references.Missing Media2012-01-15
Causes Narrative Descendant reports to output childless, spouseless children in their own generation, along with their married or parental siblings.Reports, Narrative, Jerry Bryan TrickDummyFamily-Add
DummyFamily-Delete
2012-01-15
Every direct line ancestral RIN list for one or all persons in the database with the Ahnentafel number of the last person in each line, to 64 generations, exceeding RM’s 32 gen limit on the number. Also shows all lines, not just those of parents selected for display in RM’s main Views.Ahnentafel 64 generationsAhnentafel-642012-01-13
A SQLite equivalent to RootsMagic’s own Search & Replace function on “Multimedia filenames” in the “Field to search” selection but with no programming constraintsUpdate Media PathsUpdate Media Paths2012-01-11
Removes CR/LF characters around custom fact sentences and person/family/alt name/fact notes and adds CR/LF pairs to ends of notes except the last, for first cut batch paragraphing.ParagraphingParagraph-Strip | Paragraph-Add2011-12-22
Re-orders all same day events of limited types to a natural order, e.g., Birth before DeathDates – Same Day Sort OrderSortDateSameDayOrder | SortDateSameDayOrderCustom2011-12-19
List and convert events having key properties in common to shared events. Especially useful on imports from Legacy Family Tree in which events were copied to multiple persons.Sharable Events – Find and Convert to SharedSharableFacts2 | SharableEvents-Convert2011-12-12
Makes an empty Master from current database, preserving custom fact types and source templates, Places, Master Sources and associated Repositories and Addresses, and Place/Source MediaDepopulate but keep Customs, Places, SourcesCustomDatabaseShell2011-12-10
A fact may show that it is shared with someone whose name is missing. This query lists such facts with the names of the principals and the RINs of the missing witnesses.Shared Events With Missing WitnessesTraceHeadlessWitnesses2011-12-10
There are four Internet sites that RM5’s County Check is set to call for maps and other information.County Check2011-12-10
Clones a RM4/5 database without the RMNOCASE collation so that we’re free to do what we want to the data. Even works with RM but it might trip up on non-English alphabets.Convert Database to NOCASERM#_CREATE_as_DB3_NOCASE | RM#_Copy_Data_to_DB3_NOCASE2011-12-09
Report on the role, event and owner or principal of the shared event for which a witness or sharee is no longer a person in the database.Shared Events With Missing WitnessesTraceHeadlessWitnesses2011-12-09
Marks or Unmarks members of a group according to a list; equivalent to memorising the checkboxes in RootsMagic Explorer for re-use.Named Group – Mark or Unmark List refreshGroup Unmark List Refresh | Group Mark List Refresh2011-11-27
Top page for a collection of queries for refreshing Named GroupsNamed Group RefreshRefresh – various2011-11-26
Builds or refreshes a group of persons whose lifetime probably spanned a user-defined Census Year and who had some event in the user-defined jurisdiction but not a Census fact for that year.Census Needed – Named GroupCensusNeededGroup | CensusNeededGroup22011-11-26
Provides a manual refresh for a specific Named Group, i.e., the ancestors of a specified person.Ancestors Named GroupAncestorsGroup2011-11-23
Lists all the ancestral lines for a given RINAncestors QueryAncestors2011-11-23
Produces a list of names that match or sound like specified names, similar to RM4’s NameFind.Name Find queryNameFind2011-11-21
Combo of SQLite queries and RM4 edits of Place List beats having to edit every fact/event to split a Place into Place and Place detail; otherwise, wait for the RootsMagician!Places to Place Details ConversionPlaces to Place Details Conversion2011-11-16
Complete listing of all users of media in the Gallery, more complete and navigable than what is provided in RM 4.1.2.1Media Users List QueryMedia Users List2011-11-09
About as close as we can get to a RM4 Source List Report presented in tabular form with which the results can be sorted and filtered with relative ease.Source List QuerySource List2011-11-05
This query strips extraneous Carriage Returns from the end of the Footnote sentence template for the Source Template “Vital Records (state-level, online derivatives)” which cause unwanted white space in reports.Source TemplatesFix Extra Line Feeds in Footnote2011-10-02
Converts Sources using uneditable, built-in templates to using editable copies.Source TemplatesSrcTmpltsConvert2011-10-02
Reverts Sources modified by SrcTmpltsConvert to using uneditable, built-in templates.Source TemplatesSrcTmpltsRevert2011-10-02
A Query for a To Do List in a Grid Format – using the REFN to assign status for paragraphing and census. Versions for both SQLite directly and MS Access via SQLODBC.A Query for a To Do List in a Grid FormatMyToDoList2011-07-07
Another version of a Set Living query.Another version of a Set Living querySetLivingFlag2011-07-04
A discussion of queries that can modify the Living flag.Set Living FlagSetLivingFlag2011-07-02
Variant of CopyFact2Group that copies a REFN fact to a group, substituting the target persons’ PersonID’s or RIN for that of the source person.Copy RIN to REFNCopyRINtoREFN2011-06-29
Two queries that list the test results in a format suitable for easy review and copying and pasting into other applications. RM4.1.1.4 provides no DNA report.DNA Test results listDNA_mtDNA_locationslist | DNA_Y-STR_markerslist2011-06-18
Lists Source Names along with Source Detail field names; illustrates principles applicable to Master Source fields and other XML-like columns.Source Detail View (Parsing XML)Source Detail View (Parsing XML)2011-06-06
Copies a fact/event for a person to a Named Group of persons, along with the Sources but not the Media for the Fact.Copy Fact to GroupCopyFact2Group2011-04-05
Lists files under the RM Multimedia default folder and flags those used by the RM database; helps to ensure that files are used.Scrapbook Files StatusScrapBookFilesStatus2011-03-13
Backs up the database file along with all the media files referenced by it to one file. Uses the command versions of SQLite3 (free), WinRAR ($fee) and 7-Zip (free). The 7-Zip version produces a .rmgb file that RootsMagic can restore, media and database.Backup Media with Database – RARRMfullbackup.bat2011-02-02
Backs up the database file along with all the media files referenced by it to one file. Uses the command versions of SQLite3 (free), WinRAR ($fee) and 7-Zip (free). The 7-Zip version produces a .rmgb file that RootsMagic can restore, media and database.Backup Media with Database – 7ZipRMfullbackup.bat2011-02-02
The RMSplit C# program below allows you to create a group in RootsMagic5 which contains a selected person, their ancestors, and a selected number of “leaves” (collateral lines us a given depth).SplitTreeRmsplit2011-01-03
Series of queries to list duplicate media file names in the Media Gallery and list duplicate links to items in the Gallery. Repairs a specific case of duplicate file names as an example; repairs all cases of duplicate links.Media Repair QueriesMediaRepair2010-12-14
Discussion and examples of how you can search (filter) using any SQLite manager and replace found values with revised ones using SQLiteSpy.Search & ReplaceSearch&Replace2010-11-09
This query helps you plot events from your RootsMagic database on Google Maps, Google Earth, and Bing Maps, provided there are geo-coded Places and Place Details (sites) in your database.MapEvents-KML queryMapEvents-KML2010-09-03
Have you ever wished to be able to look at all the facts in your family tree database that happened within a day’s horseride of a certain location? This adaptation of the LifeLines query helps you view your events for any geographic area in addition to looking at the lifeline of any person in your database.Geo-Lifelines QueryGeo-Lifelines2010-08-31
Lists persons with multiple spouses in descending number – may flag a data problemMultiple Spouses queryMulti-spouses2010-08-23
Lists all source names cited for a person in descending order of the count of duplicate citations – a help in finding and resolving duplicate citations after merging.All Citations & Dupes Count – QueryAllCitations+Dupes2010-07-08
Lists Individuals whose Death Year from NameTable does not match that of the date for their Death fact.Death Year MismatchDeathYearMismatch2010-06-04
Explains why an integrity check is needed and how, using PRAGMA quick_check.Check RootsMagic Database IntegrityCheck RootsMagic Database Integrity2010-05-27
Lists count of rows in each table — If not 22 tables listed, database has corruptionRMGC Tables Row CountRMGC_TablesRowCount2010-05-26
Lists all events for all persons whether in a database tree or not, including shared facts, date, fact detail, site and place, MRIN, other parties, and duplication indicator. Sorted by RIN and Sort Date. With the right SQLite manager, can filter results for one person.LifeLines – QueryLifeLines2010-03-15
Lists people who share a fact with a Principal, as well as relevant fact and Principal information.People Who Share a Fact with a Principal List – QueryPeople Who Share a Fact with a Principal2010-03-10
Lists people who share a fact with a Principal, but who aren’t in a tree in the database file, as well as relevant fact and Principal information.People Who Share a Fact with a Principal, But Who Are Not in a Tree in the File List – QueryPeople Who Share a Fact with a Principal, But Who Are Not in a Tree in the File2010-02-26
Lists duplicate name pairs with a weighted score indicating degree of match. On a large database, produces results in 4.5 minutes similar to what RootsMagic 4’s Duplicate Search Merge tool does in 45 min.Duplicate Name Search – queryDuplicateNameSearch2010-02-23
Copies all unmerged pairs from the results of RM4’s Duplicate Search Merge Tool to RM4’s ExclusionTable so that they will not reappear in subsequent runs of Duplicate Search Merge.Duplicate Search Merge DatabaseMarkNotProblem2010-02-21
Lists Sources associated with Source Templates, as well as an extension to Citation details. Free-Form templates are excluded since they’re not included in SourceTemplateTable.Source Template List – QuerySourceTemplateListWithCitationDetails2.sql2010-02-13
As ‘Database Properties’ but with more detail and pointing out possible problem areasRMGC_Properties – QueryRMGC_Properties2010-01-31
Deletes citations of non-existent Sources from database that manifest themselves in the Edit Persons screen as a citation (counted and checkmarked) but return nothing when opened.Delete Phantom Citations – QueryDelete Phantom Citations2010-01-30
Lists all citations in the database from which citations of non-existent sources (‘phantoms’) and citations for non-existent events or persons (‘headless’) can be found, along with other useful information such as all citations per source.All Citations – QueryAllCitations2010-01-30
Reports in a list most of the values found in RM4’s File > Properties report; results can be exported to a file.Database Properties List – QueryDatabase Properties2010-01-29
Decodes most of the possible formats found in RM4 Date fields of the form Da+nnnnnnnn.x+nnnnnnnn.xDate DecoderDateDecoder2010-01-21
Lists uncompleted To Do tasks for Individuals, Families and General; can be readily modified to filter or sort by surname, priority, repository, etc.RM4 To Do ListTo-Do (was To-Do4Persons)2010-01-18
Lists Individuals whose Birth Year from NameTable does not match that of the date for their Birth fact.Birth Year Mis-MatchBirthYearMisMatch2010-01-17
Returns all Places with Place Details and the Persons or Families (couples) and their Facts, including Shared events, using the Place Detail.Facts Having Place Details – QueryFactsHavingPlaceDetails32010-01-16
An optional extension to AllFacts4Persons or standalone. Having a Child as an event for the Father and the Mother.Births of children as factsPersonsBegatChildren2010-01-14
List all the Individual, Family and Shared Facts/Events for all persons in a database. A complex query using UNION ALL, COUNT() and GROUP BY, constants and NULL to assemble multiple SELECTs in one big result.Pulling Together All the Events for An IndividualAllFacts4Persons2010-01-06
List Persons with Blank Names in the Address List — (a fault that may occur in a GEDCOM import).Four Little Queriesblankname_in_addresslist2009-12-30
List of unused PlacesFour Little QueriesUnusedPlaces2009-12-30
Lists Places having Place DetailsFour Little QueriesPlacesDetails2009-12-30
List Persons with specified Surnames. — Example of creating a SQL View or Virtual Table and the explicit use of COLLATE NOCASE to override the RMNOCASE collation defined for certain fields and embedded in the RootsMagic application.Four Little Queriesselected_surnames2009-12-30

Shared Events – Sort Witnesses by Consanguinity plus Name #sharedevent

Problem

The order in which witnesses of an event (sharers other than the Principal(s) are listed on screen and in reports is set by the order they were entered. Users have no control over it.

A Workaround

The WitnessOrder field in WitnessTable controls the order of output in the display of witnesses and in narrative reports but RootsMagic 7 provides no control for it.

This script modifies the WitnessOrder values so that they are sorted:

  1. Primary: Consanguinity of witness to reference person used by the RootsMagic SET RELATIONSHIPS function
  2. Secondary: Alphabetical order of the primary name of each person, including those not in a tree.

Clearing the relationships prior to execution results in a purely alphabetical sort.

It sets negative values in WitnessOrder so inspection of the WitnessTable readily reveals those set by the script.

WitnessOrderByRelationship+Name.JPG
Example of the WitnessTable modified by the script with negative values under WitnessOrder.

WitnessOrderByRelationship+Name.sql

Relationships #relationships

This page is intended to collect some scripts in support of calculating and displaying relationships between people in a RootsMagic database.

Kinship List

This first script produces a table similar to the RootsMagic Kinship List report but it does so markedly faster for a large database. RootsMagic 7.0.11.0 took almost five minutes (290s) to calculate and display a 1900 page report for 75,000 people in a 160,000 population. SQLiteSpy 1.9.10 executed this script in 2 seconds! That’s a 145:1 improvement in speed. Moreover, an SQLite manager can sort on the columns and filter the results (features vary depending on the product) and the SQLite Views created by the script can be exploited in further ad hoc queries the user may write.

KinshipList.PNG
Screenshot of results from KinshipList.sql in SQLiteSpy sorted on the RIN column.

The script does not calculate relationships so the comparison with the RootsMagic Kinship List report is not apples to apples. Rather, it lists the relationships calculated by the RootsMagic “Set Relationships” function which added a further 50s execution time which, if included, degrades the speed advantage to a mere 5.5:1. However, it is often the case that the Kinship List is set to recompute relationships to the same person as was already done by Set Relationships. And the query can remain open in the SQLite Manager while working on the database in RootsMagic.

The columns Relate1 and Relate2 represent the distance to the common ancestor from the reference person and their relative and are calculated by RootsMagic. The script uses these to generate the Relationship phrase. They may also be used in SQLite Expert Personal as filters on the results set, e.g., limiting the people listed to those having a total distance between them of less than, say, 10. A custom query in any SQLite manager could likewise limit results.
KinshipList.sql rev 2016-07-29 sample query calculates consanguinity degree of relationship
The part of the script that generates the Views should function with any SQLite Manager. The final query that displays results with the names of people requires one with a RMNOCASE collation.

Click image for Wikipedia page on Consanguinity.
Click image for Wikipedia page on Consanguinity.

Query – All Names in Tree

Trees

Two related problems
1) Finding all names in a selected person’s tree
2) Count Trees

1) Finding all names in selected person’s tree
Does anyone have any idea how to emulate the function that is in the RM Mark Group of “Everyone in the highlighted person’s tree”? RM does this very efficiently/quickly.I break the relationship between parents and then use it to create groups of maternal and paternal lines. It works ok, but would be a lot more efficient if I could automate it using SQL.

Tree-HourglassMembers.sql This produces a subset of the full tree; not the solution… (Tom, May 8)

2) Count Trees
I sometimes forget about this function, but it is very useful in finding orphans

Hopefully I did this right. Thanks in advance
marc

Discussions & comments from Wikispaces site


MLeroux84

A solution – not perfect but workable

MLeroux84
19 September 2016 20:31:55

I finally got back to this, and can get a solution by combining the ancestor/descendent scripts with a bit of procedural logic. And adding in the missing link – the spouses of descendants. It seems to work, although it could use a lot of optimization I’m sure – it runs fairly quickly on my computer.

I tested this on my database. First, I broke the connection between my father and mother. That left me with two “trees”, my French side with 7126 entries, the Irish side with 1091 (from RM/Tools/Count Trees).

Next I created 3 tables:
tTree – contains the RIN of all individuals in the tree (final result)
tAncWork – list of RIN to traverse backwards to get the ancestors of
tDecWork – List of RIN of descendants from a list of ancestors

I populate tAncWork with my seed RIN – in my case “1″ – myself

Then I modified the Ancestor and Descendent scripts
Ancestor reads tAncWork for a list of RINs rather than a single RIN parameter and stores the result back in to tAncWork. After doing this it adds new RINS to tTree

Descendant script reads from tTree (not the most efficient way, but it was easiest) rather than a single RIN parameter and stores the result to tDecWork. The results are moved back to tAncWork and tTree. Spouse (from the familyTable) are added to tAncWork

Then the procedural part. I check tAncWork to see if there are any entries. If there are, then there are spouses that have not been counted and I run the scripts again. As expected, the new entries in tAncWork comes down quickly, and after 4 iterations I have no entries left.

That gives me one tree. Then it’s a simple matter to find the lowest Id in the remaining entries and repeat.

To generalize this, or replicate the “Count Trees” function I would have to add a “treeId” column to tTrees and populate it with the seedId for each “tree”. In my case I’m happy with the results as they are.

I don’t think it’s possible to use loops inside sqlLite but pretty simple in a procedural script. In my case I wrapped it into a VB.net app.

Thanks for the great starting point. This saves me a great deal of time – although if I think about the time invested I could probably have done this in a manual fashion for the next 80 years and still come out ahead. But it is a creative exercise that needed solving.

The table create scripts:

— Drop and recreate temp tables

drop table if exists [tTree] ;
CREATE temp TABLE [tTree] (
[RIN] BIGINT);
drop table if exists [tAncWork] ;
CREATE temp TABLE [tAncWork] (
[RIN] BIGINT);
drop table if exists [tDecWork] ;
CREATE temp TABLE [tDecWork] (
[RIN] BIGINT);

— Seed with the starting RIN

insert into tAncWork (RIN) values (1) ;

The Get Ancestors portion

— Get ancesters of list of individuals in file tAncWork
— Results stored back into tAncWork and also moved to tTree

— Based on the original script by Tom Holden ve3meo


WITH RECURSIVE
parent_of(ChildID, ParentID) AS
(SELECT PersonID, FatherID AS ParentID FROM PersonTable
LEFT JOIN ChildTable ON PersonID=ChildTable.ChildID
LEFT JOIN FamilyTable USING(FamilyID)
UNION
SELECT PersonID, MotherID AS ParentID FROM PersonTable
LEFT JOIN ChildTable ON PersonID=ChildTable.ChildID
LEFT JOIN FamilyTable USING(FamilyID)
),
ancestor_of_person(AncestorID) AS
(SELECT ParentID FROM parent_of
WHERE ChildID in (select RIN from tAncWork)
UNION
SELECT ParentID FROM parent_of
INNER JOIN ancestor_of_person ON ChildID = AncestorID)

insert into tAncWork SELECT DISTINCT AncestorID FROM ancestor_of_person, PersonTable
WHERE ancestor_of_person.AncestorID=PersonTable.PersonID AND ancestor_of_person.AncestorID > 0
;
insert into tTree
select DISTINCT RIN from tAncWork where RIN not in (select RIN from tTree) and RIN > 0
;

The get Descendants portion. This needs to run for every iteration of the get ancestors – I just split them to make it easier to debug
— get descendents of everyone in ancester table
— uses tTree as source
— puts results in tDecWork, then moves to tTree and tAncWork
— tTree contains the final result

— Based on the original script by Tom Holden ve3meo

WITH RECURSIVE
child_of(ParentID, ChildID) AS
(SELECT PersonID, ChildTable.ChildID FROM PersonTable
LEFT JOIN FamilyTable ON PersonID=FatherID
LEFT JOIN ChildTable USING(FamilyID)
WHERE
RelFather=0
UNION
SELECT PersonID, ChildTable.ChildID FROM PersonTable
LEFT JOIN FamilyTable ON PersonID=MotherID
LEFT JOIN ChildTable USING(FamilyID)
WHERE
RelMother=0
),
descendant_of_person(DescendantID) AS
(SELECT ChildID FROM child_of

WHERE ParentID in (select RIN from tTree)
UNION –ALL
SELECT ChildID FROM child_of
INNER JOIN descendant_of_person ON ParentID = DescendantID)
insert into tDecWork SELECT distinct DescendantID FROM descendant_of_person, PersonTable
WHERE descendant_of_person.DescendantID=PersonTable.PersonID AND descendant_of_person.DescendantID > 0 ;

— Move results to tAncWork

delete from tAncWork ;
insert into tAncWork
select distinct RIN from tDecWork where RIN not in (select RIN from tTree) AND RIN > 0
;

— Move results to tTree

insert into tTree
select distinct RIN from tAncWork where RIN not in (select RIN from tTree) and RIN > 0
;

— Add their spouses to tAncWork to prapare for next itiration

insert into tAncWork
select distinct motherId from familyTable
where fatherId in (select RIN from tTree)
and motherId > 0
and motherId not in (select RIN from tTree)
UNION
select distinct fatherId from familyTable
where motherId in (select RIN from tTree)
and fatherId > 0
and fatherId not in (select RIN from tTree);

— Finally, clean out decWork

delete from tDecWork ;


ve3meo

Inline comment: “1) Finding all names in a selected person’s tree‍”

ve3meo
04 September 2018 01:25:23

ve3meo May 8, 2016

Good question for which I have no ready answer. It was difficult to do lineage before SQLite gained recursive query functionality. Doing recursions in SQLite does not come naturally to me. Your goal requires not only ancestors and descendants but also collateral lines. While it does not require kinship results which should make it easier, I confess I do not have a strategy in mind.
MLeroux84 May 8, 2016

Thanks. recursive SQL is not that easy for me either. I was thinking that it would not be that difficult, since it executes so quickly from the RM application. I haven’t seen it take very long to do “Count Trees” – well under a second for 4K names, so I was hoping that this was something obvious that I missed.


ve3meo

Inline comment: “2) Count trees”

ve3meo
04 September 2018 01:26:30

ve3meo May 8, 2016

This one mystifies me, too. It seems that it needs to be a recursion of “Finding all persons in a given person’s tree” iterating through all those remaining persons after subtracting those trees that have been counted.
MLeroux84 May 8, 2016

I wouldn’t think that it was iterating through remaining names, just because of the challenge of recursing through each name. Given the performance I’m seeing (albeit on a machine with reasonable performance) It would seem more likely that there is a way of identifying a node of each tree, then counting from there.

That said, since I haven’t been able to solve the first part of the problem it is all pure speculation


ve3meo

Inline comment: “produces a ‍subset‍ of the full tree”

ve3meo
04 September 2018 01:27:36

ve3meo May 8, 2016

This script includes all the ancestors of the starting person and their descendants but the full tree includes all the descendants of all the ancestors of all the descendants of all the ancestors of…
MLeroux84 May 8, 2016

Thanks, Tom. I played with this a bit today. I’m hampered by my lack of knowledge/experience with recursive queries. I get a bit further (but not very much) than you did by including spouses in the ancestor query. I’m thinking that each iteration needs to include siblings and spouses as seeds for the next iteration, but have not been able to find a way that seems to work.
I’ll keep plugging at it – it has gone from something “I’d like to do” to “It’s really bugging me now”

Inline comments


ve3meo

Comment: Good question for which I have no rea…

ve3meo
08 May 2016 17:31:19

Good question for which I have no ready answer. It was difficult to do lineage before SQLite gained recursive query functionality. Doing recursions in SQLite does not come naturally to me. Your goal requires not only ancestors and descendants but also collateral lines. While it does not require kinship results which should make it easier, I confess I do not have a strategy in mind.


MLeroux84

MLeroux84
09 May 2016 02:01:42

Thanks. recursive SQL is not that easy for me either. I was thinking that it would not be that difficult, since it executes so quickly from the RM application. I haven’t seen it take very long to do “Count Trees” – well under a second for 4K names, so I was hoping that this was something obvious that I missed.


ve3meo

Comment: This one mystifies me, too. It seems …

ve3meo
08 May 2016 17:31:20

This one mystifies me, too. It seems that it needs to be a recursion of “Finding all persons in a given person’s tree” iterating through all those remaining persons after subtracting those trees that have been counted.


MLeroux84

MLeroux84
09 May 2016 02:07:11

I wouldn’t think that it was iterating through remaining names, just because of the challenge of recursing through each name. Given the performance I’m seeing (albeit on a machine with reasonable performance) It would seem more likely that there is a way of identifying a node of each tree, then counting from there.

That said, since I haven’t been able to solve the first part of the problem it is all pure speculation


ve3meo

Comment: This script includes all the ancestor…

ve3meo
09 May 2016 03:58:26

This script includes all the ancestors of the starting person and their descendants but the full tree includes all the descendants of all the ancestors of all the descendants of all the ancestors of…


MLeroux84

MLeroux84
09 May 2016 23:11:22

Thanks, Tom. I played with this a bit today. I’m hampered by my lack of knowledge/experience with recursive queries. I get a bit further (but not very much) than you did by including spouses in the ancestor query. I’m thinking that each iteration needs to include siblings and spouses as seeds for the next iteration, but have not been able to find a way that seems to work.
I’ll keep plugging at it – it has gone from something “I’d like to do” to “It’s really bugging me now”

Events – Move Description to Note #events #notes

This script moves the entire contents of event descriptions of a selected fact type to the Note for that event. If there is already content in the Note, the Description value is appended as a new paragraph.

To select a Fact Type, you must use a SQLite manager that supports SQLite run-time variables, such as SQLite Expert Personal (SQLiteSpy does not). When the query is run, a box will pop out prompting for the entry of the Fact Type’s abbreviation. This is the label as seen in the Edit Person screen. By entering the wildcard character (%), all Fact Types having the Description enabled will be processed.

EventDescriptionToNote-Move.sql
— adapted from Events – Move Short Note to Description