Maps – markers proportional to number of events #googlemaps #geocoding

2021-11-22 Compatible with #RM8

An earlier script, Places – Frequency of Use, produced a result set that counted the number of events in each Place in a RootsMagic database. This script builds on that query to generate a comma-delimited data set suitable for uploading to GPSVisualizer.com to generate a map with circle markers that are proportional in size to the number of events.

PlaceFreqGPSVisualizerMap.PNG
Example of a proportional symbol map generated from data extracted by Place_Frequency_GPSVisualizer.

The script produces a result set similar to the screenshot below from SQLiteSpy. Place_Frequency_GPSVisualizer.sql.png
Select any row, then Ctrl-A to select all and Ctrl-C to copy all selected to the Windows Clipboard. You can choose to paste directly into the GPSVisualizer form or into a text editor (NotePad) and save it to a file with the .csv extension for upload and reuse.
GPSVisualizerUploadForm.PNG
Place_Frequency_GPSVisualizer.sql

Spouse Order – Global Sort by Marriage or First-Birth SortDate #spouse

When the person highlighted in Pedigree View or Family View is linked to more than one spouse, RootsMagic 7 lets you select the one to be shown from the dropdown Spouse list. This list also offers a “Rearrange spouses” control which sets the order of the spouses in the dropdown list and the order in which spouses are displayed in the Descendant and Timeline Views, the Edit Person screen and in certain reports. The Rearrange Spouses dialog allows both manual ordering and “Sort by Marriage Date”. This control operates only on the spouses of one person at a time. A wish has been expressed for a global Sort by Marriage Date. Furthermore, an emigre from Family Tree Maker pointed out that it also factors in the birth date of the first child in the case of an unwed couple. These requests may never see the light of day within RootsMagic but here is a script that does the global job in the interim.

Some words of caution. For spouses without marriage events or children, neither the script nor the current RootsMagic function can do anything but sort them to the top of the list. And this script will override sequences arranged manually through RM’s Rearrange Spouses, if there are two or more of the spouses with either a marriage event or child birth event.

Developed using SQLiteSpy and should require no special extensions so it should run with any recent SQLite manager.

SpouseSort2.sql
2016-01-12 Superseded SpouseSort.sql to factor in birth/christen date of first-born children of unwed couples in the absence of a marriage event.

As always, test it on a copy of your database or make a backup first.

Roles-Sort Alphabetically #roles #facttypes

RootsMagic 4-7 (current) list the user-added roles in the Edit Fact Type dialog in the order that they were created, after the builtin roles such as Principal and Witness. For fact types to which many roles have been added, it is desirable that they be sorted alphabetically. While this should easily be done within the program for display purposes and has been requested repeatedly over years, until such an enhancement is provided here is a script that uses a brute force method to rearrange the rows in the data table so that the role names are presented alphabetically.
RoleSortBeforeAfter.png
As far as I can tell, the script is harmless. There is no need to use RootsMagic’s Rebuild Indexes after its execution but it would be advisable to test it on a copy of your database or make a backup before applying it. It was developed using SQLiteSpy but uses no esoteric functions so it should run with any SQLite manager that supports a fake RMNOCASE collation extension..

Caveat: If a future version of RM increases the number of builtin roles to more than 58, every instance of 58 in the script must be revised accordingly else those builtin roles above the RoleID of 58 will be mixed up with a user-added role after a drag’n’drop or GEDCOM transfer to another RootsMagic database..

Tip: If you want the role names to appear in an order sorted by your preference, rename them prefixed by a number, e.g.:

  1. with Parents
  2. with Father
  3. with Grandparents
  4. Other

If you have more than 9 roles, then use double digits:
01 with Parents
02 with Father

RoleSort.sql

Group – Persons with Duplicate Events #namedgroup #events #date #duplicates

2023-02-19 Version for #RM8 added.

  • Creates a Named Group in the open database with the name “SQL: Duplicate Events” if one does not already exist
  • Removes all members from that named group
  • Populates that named group with persons having “duplicate” events

Duplication is declared if a pair of events:

  • (have dates for which the span of calendar years (ignoring month and day) is less than a parameter (2 in the script) OR
  • do not have dates but are fact types that use dates) AND (are both set or both not set as Primary events)

Duplication is resolved in RM by:

  • deleting an event (you may want to incorporate an alternative date in the date or notes of the remaining event) OR
  • setting one event only of the pair as Primary

To refresh the named group’s membership list to reflect corrections you may have made to resolve duplication of events, rerun the query to regenerate the RootsMagic group table and then, in RootsMagic, select a different group and then re-select the “SQL: Duplicate Events” group. You should see that names have disappeared from the group, provided you have corrected all the duplicate events for each person you worked on. The Quick Groups feature in RM7 and the Groups list in RM8 allow you to unmark a member from the group right from the Edit Person screen making it unnecessary to rerun the script unless you missed doing so.

To vary the parameter controlling how sensitive the query is to date differences, change this line at the end of the script:

WHERE ABS(SUBSTR(Date1,4,4) - SUBSTR(Date2,4,4)) <2 -- this controls how tight the year match must be.
-- in future, the date comparison could call on a date calculator that could compute the difference in days
 
)
; --END of SCRIPT

A setting of “<2” permits two consecutive years which allows Dec 31, 1900 and Jan 1, 1901 to be considered duplicates as it does for Jan 1, 1900 and Dec 31, 1901. A setting “<1” requires the events to be in the same year.

Group-PersonsWithDuplicateEvents.sql rev 2018-07-07 bug fixed when no other existing group

RM8 Version 2023-02-19

Nickname Manipulations #nicknames

Why?

RootsMagic has three different storage mechanisms for Nicknames:

  1. Nickname field in each Name record
  2. In quotation marks or parentheses in the Given field of each Name record
  3. As above in a Name record of type “Nickname”

Different outputs from RootsMagic have differing usage of these. Box charts cannot use Alternate Names and only include the primary name Nickname field if Prefix and Suffix are also included. Sentence templates for facts other than the Alternate Name type cannot access Alternate Names, cannot parse the nickname embedded in the primary Given field but can use the primary Nickname field. So there are outputs where it may be desirable that nicknames be embedded with the given name and others where it should not be. And as a database is built over time from a variety of sources, there may be inconsistency in the way nicknames are stored within the database.

This page offers a set of scripts that can manipulate where the nickname values are stored, globally and quickly, to optimise for a desired output and to make consistent.

Combine | Split Nickname and Given Names

NickGivenSplitCombine.png
These two scripts can swap the nickname between the NickName field and the Given Names field. The splitter works only with quotation marks delimiting the nickname in the Givens field; the combiner puts the nickname in quotes in the Givens field. You can use RootsMagic Search and Replace function on the Given names to convert all instances of parentheses to double quotation marks.
NickGivenSplit.sql
NickGivenCombine.sql

Ambivalent Nicknames

This script returns Record Number and Names of Persons who have both an embedded Nickname and a value in the Nickname field. These cannot be processed by NickGivenCombine.sql and NickGivenSplit.sql and should be resolved in the database with one or the other but not both.

Includes those having parentheses to denote nickname in the Given field instead of quotes; parentheses not supported by other scripts in this series.

NicknameAmbivalent.sql

Conflict between Primary and Alternate Nicknames

This script returns the Record Number and Nicknames of Persons who have both an Alternate Nickname and a Primary name with an embedded nickname or non-empty NickName field and the nickname values in the corresponding fields do not match. Use RIN to find the person in RootsMagic Explorer to revise as needed.
NickAltNameConflict.sql

UNDER CONSTRUCTION
-possibly converters between the nickname fields of Primary Name and Alternate Name of type Nickname but this would be complex.

Media Users List Query #media #rm8

Update

As of RootsMagic 8 Preview 7.9.180, the traceability of where an image is used has changed little from what was described below for RM4. The need for an outboard query such as this is as high in early 2021 as it was in 2010. With some changes in tables from RM7, the Media-UsersList4.sql query does not run on RM8. Pat Jones revised it and applied a technique she describes at Common Table Expressions – The Building Blocks of SQL to streamline it:

Media-UsersList4-RM8.sql

For a version that works with databases from RM5 to RM7:
Media-UsersList4-RM7.sql 

Introduction

RootsMagic 4 falls short in handling media in several ways. One serious one is its inability to report fully all the ‘owners’ of the media, i.e., its terminology to describe what person, couple, event, source, citation, place or place detail to which a media item is attached, in sufficient detail that the attachment can be found. Trace-ability is essential to be able to make changes to an attachment’s properties, i.e., its caption, description, reference number, date, sort date, primary and scrapbook inclusion flags. RM4’s Report > Lists > Multimedia List does not even report usage by Sources or Source Details and fails to report the Place to which a Place Detail, having media attached, belongs. Similarly, the Media Gallery > Tools > “Show where this media is used” results are incomplete for media attached to Source Details or to Place Details, giving little or no clue to how you would navigate to the citation or Place where you could then drill down to the Media and edit the caption or other properties.

Here’s an example of what I mean:

Owner according to Media Gallery > Tools > “Show where this media is used”

Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete) (citation)

How can one possibly find the citation without at least the RIN or name of the person?

Users according to Media-UsersList query

(citation) WICKENS, Martha Jane "Jenny"-208 : RESIDENCE (FAMILY) ca 1891 : HOLDEN, Matthew Edwin-149 citing Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete)
(citation) HOLDEN, Matthew Edwin-149 : RESIDENCE (FAMILY) ca 1891 : WICKENS, Martha Jane "Jenny"-208 citing Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete)

Both RM4 and the query show that the media file is attached to a citation but the new query provides explicit guidance to the citation via not only the name and RIN of the person but also the fact name and date so that the citing fact can be quickly found and its Citation Manager opened. The source name is shown (that’s all the Media Gallery shows apart from the caption and full filename) so it can be readily selected from the Citation Manager and the Source Detail Media Album opened therefrom and the media item selected by file name for the editing of its properties. That is the only way RM4 lets you get to the item’s properties; obviously, a road map is necessary. This query shows the route.

Download

Media-UsersList.sql – original query, easier to modify, fast on smaller databases, bogs down on large ones
Media-UsersList3.sql RMtrix_tiny_check.png – fast even on very large databases, larger sql file, harder to modify at the base SELECTs.
Media-UsersList4.sql – as ver 3 but revised for RM5 and 6; prior versions work with RM4.

Screenshots

With Thumbnail Images

Using SQLite Expert Personal, make sure the following setting is checked for the thumbnail image to be shown:

Tools > Options > Data:

  • Results-Display results in grid
  • Grids-Show images
Media-UsersList_Screenshot_from_Expert.PNG
SQLite Expert Personal can display the thumbnail images stored in the database. A subset of the available columns is displayed to reduce clutter. Unfortunately, column widths are not remembered on re-executing the same query. Also, sorting on column heading does not work unless the ORDER BY clause at the very end of the query is deleted.

Without Thumbnail Images

Media-UsersList_Screenshot_from_Spy.PNG
Screenshot from SQLiteSpy which has no image display capability but does remember column widths within the same session and supports sorting on column regardless of an ORDER BY in the query.

Discussions & comments from Wikispaces site


ve3meo

 

Comment: “Download”

ve3meo
03 September 2018 21:38:03

ve3meo Nov 14, 2011

This query can bog down on very large databases because it produces a very large set of intermediate results that must be stored in memory. It is being (slowly) revised to produce many small sets of intermediate results so that it will be able to handle large databases efficiently. This is not a trivial re-write.
ve3meo Nov 14, 2011

Re-write done: Media-UsersList3.sql. On a large database that was taking 37 to 120 seconds to display results, the time has dropped to 1-5s, depending on the SQLite manager, what else is going on, how much memory is available.The key was to structure it to minimise the size of intermediate results that are then JOINed with another result set.

Citations Invisible – Reveal and Revert #citations #sources #ftm2014 #ftm2012 #ancestrycom #gedcom

With the flurry of users of Family Tree Maker looking to migrate their database to RootsMagic, an outcry has gained volume because of the disappearance of some citations. This problem was described here in 2012 in Citations Invisible Revealed which offered two scripts, both of which are available in the RMtrix utility, one to find such citations for the Primary (Preferred Name), the other to convert them to citations for the Person. That was an irreversible conversion. In the hope that RootsMagic will eventually support citations for the Primary (Preferred) Name, this pair of scripts provides both conversion and reversion. As of 2015-12-24, RootsMagic is expected to issue an update that will do a similar conversion and an upgrade that will support citations for the Primary (Preferred) Name and revert these conversions. However, it will use a different value in the flag to make the citations reversible than was used in the initial scripts, necessitating these revised versions to be compatible.

Citations, Invisible – Convert to Personals + Flagged.sql rev 2015-12-24 Flags field now set bitwise on 1st bit instead of to 7
Citations, Invisible Revealed – Revert – Bitwise.sql 2015-12-24 Unsets 1st bit of Flags field instead of setting to 0

Caveat! To make it reversible, the converter uses the apparently unused Flags column in the CitationTable. The initial converter dated 2015-12-19 set this value to 7, corresponding to the original OwnerType value for the Citation Table record; the value that RootsMagic will use for this purpose will be the 1st bit set to 1, allowing the Flags field to be used for many bitwise flags. I cannot predict what the consequences might be for those records having a flag value of 7 set by the initial converter. If you used the now-deprecated, initial version of Citations, Invisible – Convert to Personals + Flagged.sql, it would be advisable to run this complementary script to revert them to the primary Name fact: Citations, Invisible Revealed – Revert.sql Then follow up with the revised converter above or wait for the RootsMagic update to do the same thing.

James Clouse posted this video to the RootsMagic Users Facebook Group showing how to use SQLiteSpy to open your database and run the conversion script. Click on the settings and select HD to avoid fuzziness.

Earlier Queries by Date

Pages since RootsMagic 7 released

2018-07-31 Events – Add to all Children a Parent event shared with parents added
2018-07-22 GEDCOM & DnD transfer losses added
2018-07-13 File Question added (re: .DAT files in C:\Users\username\AppData\Roaming\RootsMagicAMT)
2018-03-19 Migrating from Wikispaces added
2018-01-28 TreeShare – Link Pasted Ancestry Sources added
2017-12-27 AllCitations.sql Error added
2017-12-16 Possible Orphaned Records in the Event Table added
2017-12-13 Database is locked added
2017-12-12 Exporting Data added
2017-11-21 Query is taking a very, very long time added
2017-11-11 SQLite Expert Personal Edition Version 5.2.2.240 (x64) problem added
2017-10-25 Where is this data? added
2017-10-18 Removing media from one person added
2017-10-17 Where is the data stashed added
2017-09-15 Alternate Names List added
2017-09-13 Query to Add Parents Events to RM Database added
2017-09-03 Identifying Events with Individual Sentence Customization added

2017-08-06 Database Design Spreadsheets added. Updated workbook to cover RM 7.5 and converted to Google Sheets.
2017-07-21 Ancestry TreeShare – Impact rev. Added: Disconnect but preserve Ancestry Sources links for next Upload
2017-07-09 Groups – Extract most everything for one to a new database added
2017-07-05 TreeShare – Rename Cryptic Filenames for Citation Media added
2017-07-01 Search – wayfinding from data tables to RootsMagic screens rev script for new table for TreeShare
2017-06-28 Ancestry TreeShare – Impact added; ongoing revs. TreeShare released in RM7.5 that day.
2017-04-20 Places – Count People and Events for a Group added
2017-04-19 Source Templates – Convert Builtins to Editable, Split, Import added
2017-04-19 Navigation Sidebar revised – removed Poll, added Nav widget and Tag Cloud
2017-04-17 A Sample Query Created with Views rev with addition of script Events-CitationsMissing-UsingViews.sql
2017-04-06 REMATCH to FamilySearch ID in REFN fact added
2017-03-30 Sources – Unreverse Author Names added
2017-03-21 Source List Query rev MasterSources.sql to show unused Master Sources
2017-02-27 Places – Delete Unused added.
2017-02-25 Names – married name in death sentences added. Adds or replaces custom local sentence to use husband’s surname..
2017-02-20 SpouseID Invalid added. Addresses cause of empty Family View symptom.
2017-02-11 Merge Duplicate Single Parent Couples added. Rev 2017-02-14 corrected bug
2017-02-11 Births – Add from Christening or Baptism added
2017-02-05 Names – Delete Duplicate Alternate Names of Type ‘Married’ added.
2017-01-03 Group – Population statistic added. Lists group names and number of people in named groups.
2016-11-20 WebTags – convert to Notes et al edited for ease-of-use, speed and reusabaility
2016-09-05 COPY FamilySearchID to REFN fact added. One use is to reliably match duplicate people for merging.
2016-08-18 A Sample Query Created with Views added. Demonstrates a modular approach to building a complex query.
2016-08-16 Reports, Narrative, Jerry Bryan Trick rev. Ver 2 reduces the instances of excessive white space, a collateral effect of the trick.
2016-08-12 Reports – New Paragraph for General Note added. Improved format for narrative reports.
2016-08-02 Quick Start for Dummies added. Know nothing about SQLite? Start here…
2016-07-30 Color Code by Consanguinity Degree added.
2016-07-29 Shared Events – Sort Witnesses by Consanguinity plus Name added. Sorts sharers of event by consanguinity, then by name.
2016-07-29 Relationships rev. Now calculates consanguinity degree.
2016-05-08 Events – Move Description to Note added. Moves the entire contents of event descriptions of a selected fact type to the Note for that event.
2016-05-08 Query – All Names in Tree added. Lists all those in the hourglass of the defined person’s tree.
2016-04-22 Source Templates – Merge Duplicates rev. to make insensitive to case and space character.
2016-04-05 Reports – Point Form Narratives Setup version 2.
2016-03-28 Maps – Geo-Pedigree, plotting your ancestors’ birth places started but aborted in favour of RootsMapper.com
2016-03-28 FTM import – restore Event description from Place Details added. Move the content from each event’s Place Detail to the event’s Description field.
2016-02-12 Statistics added. A start; reports the male and female having the maximum children total with all spouses.
2016-02-07 Media – Bulk Rename and Relink added. Converts structured media filenames to another structure and relinks them.
2016-02-04 Language other than English added. Demonstrates current limitations, working in French.
2016-02-01 Reports – Point Form Narratives Setup added. Makes the Narrative format less verbose, easier to scan.
2016-01-29 Maps – markers proportional to number of events added
2016-01-21 Relationships fast Kinship List added
2016-01-17 Facts – Split Shared to Individual edited: added version without RIN displayed
2016-01-16 RIN MRIN – add Reference Number fact to all persons added
2016-01-10 Spouse Order – Global Sort by Marriage or First-Birth SortDate added
2016-01-08 Source Citation Sort added
2016-01-07 Roles – Sort Alphabetically added
2015-12-31 Nickname Manipulations added
2015-12-19 Citations Invisible – Reveal and Revert added (of interest to migrants from FTM)
2015-11-24 Recursive Queries – Ancestors and Descendants rev (gender filter for genetic lines)
2015-10-18 Children – Set Lineage to Birth added (so RootsMagic ancestor reports follow birth lines)
2015-03-30 TMG to RootsMagic Migration added (index and summary of useful pages for the TMG emigre)
2015-02-16 Marriages, Length of added (answers how long each marriage lasted)
2015-01-25 Living Flag – Set Globally added (can correctly set/clear living flag without Death fact in most cases)
2015-01-23 Children – Needing Manual Arranging added (lists families in which one or more children lack Birth or Christen facts)
2015-01-22 Children – Set Order by Birth SortDate added (batch sorting of children in birth order)
2015-01-21 Places – Conversion of Place Details to Places added (for when you need to export to other software)
2015-01-05 SQLite Managers for Mac OS added
2014-12-11 Sources – Copy Repository Name, Address from Repository List added; rev 14 Dec.
2014-11-25 RootsMagic 7 added

Pre-RM7 Queries

Most work with RM7 databases; most scripts since RM7 are applicable to RM6 but are less so with the older versions.

2014-12-11 Sources – Copy Repository Name, Address from Repository List added; rev 14 Dec.
2014-11-25 RootsMagic 7 added
2014-11-24 Reports – Concordances for Indexes added
2014-11-20 Fact Type – Convert Census to yyyy Census and back added; support for shared events added 2014-11-21!
2014-10-30 Names – Move Prefix to empty Suffix added
2014-10-29 Names – Edit NameTable with SQLite Expert added. A great starting point for beginners!
2014-10-28 Search – wayfinding from data tables to RootsMagic screens added
2014-10-23 Search – Find Almost Everywhere added separate scripts for Build and Search
2014-10-21 Facts – Change Fact Type added;.support for shared events added 2014-11-22!
2014-10-16 Search – Find Almost Everywhere added
2014-10-14 Database – Copy Master Lists to Shell revised to include File Options
2014-09-30 TMG-RootsMagic Split Citation Details revised to support up to CD9
2014-09-28 TMG-RM Fact Sentence Tweaks added
2014-09-25 Events – Move Short Note to Description added
2014-09-24 TMG-RM Check for False Spousal Event Sentences added
2014-09-23 TMG-RM Convert TMG_ID to Record Number added
2014-09-23 TMG-RM Convert TMG_ID to Reference Number added
2014-09-21 TMG-RootsMagic Split Citation Details added, supported CD1, CD2 only
2014-09-21 TMG-RootsMagic Sentence Variables & Format Codes added
2014-08-31 RMpi GEDCOM Pre-Import Tweaker for RootsMagic added
2014-08-23 Notes Invisible Revealed added
2014-08-22 GEDCOM & DND Event Description Length Anomalies & Bugs added
2014-07-17 Ancestry.com Sync started
2014-04-15 Group – Ancestors added
2014-04-15 Group – Descendants added
2014-04-14 Recursive Queries – Ancestors and Descendants added
2014-04-13 Ancestors Query – Recursive added
2014-03-18 Citations – Delete Duplicates added
2014-03-05 Sources – Adventures in Extreme Splitting clarified results of transfer to FSFT
2014-02-09 RMNOCASE – Nunicode Extension added
2014-01-24 Group – Unconnected Persons added
2014-01-21 Facts – Reference Numbers for person, spouse and parents. added
2013-12-24 Privatize Living added
2013-12-17 Source Templates – Versatile Free Form Hybrids added
2013-12-14 Question with respect to Changing a Master Source to a Different Source Template added
2013-11-25 An Example of Using SQLite Views added
2013-11-15 RMNOCASE – faking it in SQLite Expert, command-line shell et al C source code added
2013-10-21 Places without Geo-codes for Named Groups added
2013-10-19 MapEvents-KML query updated with Bing and Geocommons examples
2013-10-15 Facts – Add custom MRIN event to each family added
2013-10-06 How to Query RootsMagic added Example – Change fact type globally
2013-09-29 Source Templates – Set Quotation Punctuation to ‘UK’ or ‘Logical’ Placement added
2013-09-24 Source Templates – Revision Utilities added
2013-09-23 Media – Set Primary Photo for Persons added
2013-09-02 Group – Persons with Duplicate Events added
2013-09-01 Group – Persons with Text Dates added
2013-08-07 Sources – Merge Duplicate Masters added
2013-07-30 Source Templates – Merge Duplicates added
2013-07-27 RMNOCASE – faking it in SQLite Expert, command-line shell et al added
2013-07-01 Rebuttal by Jeff La Marca added
2013-06-26 RMNOCASE – faking it in SQLiteSpy bug cleared
2013-06-24 RM Infrastructure to support FSFT added
2013-06-18 Sources – Adventures in Extreme Splitting added
2013-06-07 Query for Sources Without Media added
2013-04-08 Source Templates – Field Count added
2013-03-30 Places – first name exploit for improved narratives added
2013-03-27 ReportNon_proven added
2013-03-25 Places – Frequency of Use added
2013-03-23 Places – Recombine Fractures from FTM 2012 added
2013-03-14 Phantom Spouses – Unnamed and childless added
2013-03-04 WebTags – convert to Notes et al added
2013-02-17 Place Names – parse and recombine added
2013-02-14 Events – Merge started (response to Need a Starter Idiot Guide)
2013-01-24 Date Last Edited added
2013-01-23 Color coding Snapshot and Recall added (lost? 2019-01-27 TH)
2013-01-21 RMtrix updated with 20 additional functions
2013-01-08 Tools – Problem Search – Spouse Order Questioned added
2013-01-07 Facts – Sort Order Problems added
2012-12-31 Dates – SortDate Algorithm updated and added SortDateEncoderDev.sql
2012-12-28 MediaTags – Copy Shared Facts Media To Sharee Personal and MediaTags – Delete Personal Having Fact Duplicates
2012-12-19 Facts – Split Shared to Individual
2012-12-12 Query Menu and chronological list updated.
2012-12-10 WebTags – from Ancestry.com and FTM, WebTags – Consolidate, WebTags-DeleteDuplicates.sql
2012-12-04 Media Type Reset
2012-12-03 Collected utilities by R. Steven Turley.
2012-12-02 Database documentation revised to RM6.
2012-11-19 RootsMagic 6 released.

Citations Invisible Revealed #citations #ancestrycom

N.B. This is now the background page for a pair of revised scripts that supersede the conversion script here and make the process reversible. See Citations Invisible – Reveal and Revert.

First reported by Paul Harris in the RootsMagic Forums in the discussion Unused Citations after Import, we have concluded that there is a bug in RootsMagic 5.0.4.1 and, possibly, earlier versions that suppresses or fails to include a certain type of citation from appearing in key user screens and reports. Yet these citations are generated by RootsMagic, can be seen in the Source List report and survive a drag and drop transfer to a new database.

Citations, Invisible – List.sql RMtrix_tiny_check.png This query will show whether there are any such ‘invisible’ citations in a RootsMagic database.
Citations, Invisible – Convert to Personals.sql RMtrix_tiny_check.png This query will convert these ‘invisible’ citations to a type that is visible in the RootsMagic program.

The ‘invisible’ citations covered by these queries are for the Primary Name of a person and are converted to a Personal citation to become visible. We think an Ancestry citation that “provides evidence for Name” is exported as a citation for the “Preferred” Name, when there are Alternate Names, not for the Person. If there are no Alternate Names, we surmise that Ancestry exports the citation against the Person (top level in GEDCOM, Name is lower); these are visible. Citations for Alternate Names are visible and correctly placed in the Edit Person screen. Since there is no other place for a Primary Name citation to go than beside the name at the top of the Edit Person list and since that is where Personal citations do get flagged, then it is reasonable to convert the ‘invisible’ citations to Personal ones.

On import, RootsMagic assigns an OwnerType = 7 in the CitationTable for citations for Names, be they Preferred or Alternate, signifying that the corresponding CitationTable.OwnerID points to a unique NameID in the NameTable. Each unique NameID has a set of names for a Person in the PersonTable whose unique PersonID corresponds to the NameTable.OwnerID. For a given NameTable.OwnerID (or Person), there should be but one record with the IsPrimary column set to 1 – this is the Primary Name; all other records in the set have IsPrimary = 0, signifying they are Alternate Names. The List query lists all citations of Type 7 that point to a Primary Name.

For Personal citations, OwnerType = 0 in the CitationTable, signalling to RootsMagic to match the CitationTable.OwnerID with PersonTable.PersonID and that this is a Personal citation.

The Convert to Personals query replaces the CitationTable.OwnerID with the PersonID owning the NameID pointed to by the citation and sets OwnerType to 0 for only those citations for a Primary Name (OwnerType = 7, linked NameTable.IsPrimary = 1 or True).

Dates – Same Day Sort Order #date #sortdate #update

#RM10 compatible

Problems

Some RootsMagic users bemoan the fact that events having the same date are displayed and reported in an unnatural or illogical order, having more to do with the order they were entered. Burial before Death or Christen before Birth are often mentioned. RootsMagic does provide the user with the option to modify the SortDate date to establish the desired order.

RM’s proprietary SortDate is lost in an export to another program; any import into RM from anything other than a RM database or an RM generated GEDCOM with the Extra Details (RM specific) box checked will be assigned SortDates matching the event date resulting in the probable loss of the desired order. So manual control over the SortDate is of no comfort to those who move their database between different programs to take advantage of their respective strengths. An enhancement to provide better results for the order of same day events has long been on the RootsMagic Wish List.

Workaround

Meanwhile, here are two SQLite queries that can re-order all same day events of limited types in seconds with results as shown in the screenshots below. They both exploit the SortDate feature that values of the form date-n, where n is an integer from 1 to 4999, are sorted in ascending order.

Dates-SameDaySortOrder.png
Before and after the query was applied.

SortDateSameDayOrder.sql

SortDateSameDayOrder.sql, currently addresses the following in the order listed: Birth, Christen, Baptism, Death, Cremation and Burial. Others could be added. Should work with any SQLite manager on both RM4 and RM5 databases. Currently deprecated because it changes other facts on the same day; left here because its comparative simplicity may help to understand its successor below.

/* SortDateSameDayOrder.sql
   2011-12-19 ve3meo
   Alters SortDates of Birth, Christen, Baptism, Death, Cremation and Burial
   to a natural order when any pair or more occur on the same date.
   Could be extended to order other facts also. SortDates are effectively assigned
   (by arithmetical offsets) an absolute suffix -1, -2, ... related to the FactType.
   Affects only those events whose SortDates correspond to the Fact Date, as computed
   by a Date encoding algorithm. The algorithm does not handle Date modifiers so not all
   Event dates are handled, e.g. "Bef 1960".
   2011-12-20 order of Cremation and Burial corrected
*/
 
UPDATE EventTable
SET SortDate = SortDate-6692012023*(EventType
  IN (1,3,7,2,4,5))  -- list of FactTypes we want to sort, in no particular order except this corresponds to the desired order
  +1048576*(SUBSTR('1426503',EventType,1)-1) -- the substr maps the FactType to its order
WHERE EventID
IN (SELECT EventID FROM EventTable
    INNER JOIN
    (SELECT -- matching dates
     SortDate, OwnerID, COUNT()-1 AS Matches FROM EventTable
     WHERE OwnerType = 0 AND EventType IN (1,3,7,2,5,4)
     AND
     SortDate =   -- equals encoded event Date (if not a match, suggests that user has modified SortDate so don't touch it)
       (CASE
        WHEN DATE LIKE '.%'
        THEN 1
        ELSE Substr(DATE,3,5) END
        +10000
        )*562949953421312
        + Substr(DATE,8,2)*35184372088832
        + Substr(DATE,10,2)*549755813888
        + 17178820620
      GROUP BY SortDate, OwnerID, OwnerType
     )
     USING (OwnerID, SortDate)
     WHERE Matches
    )
;

SortDateSameDayOrderCustom.sql

SortDateSameDayOrderCustom.sql RMtrix_tiny_check.png
Goes to the next level, supporting the ranking of any number of facts, be they standard or custom. It also corrects an oversight in SortDateSameDayOrder.sql and does not touch unranked facts on the same date. Requires the use of a fake RMNOCASE collation or possible revision with COLLATE NOCASE phrases where FactTypeTable.Name is used. Because SortDate supports up to date-4999, this query could be modified to have a multiplier of the rank, thus leaving room for other events on the same date to be manually added with a SortDate lying between those assigned by the query, without having to revise the existing SortDates.

/* SortDateSameDayOrderCustom.sql
   2011-12-20 ve3meo
   Alters SortDates of any set of Fact types
   to a natural order when any pair or more occur on the same date.
   Could be extended to order other facts also. SortDates are effectively assigned
   (by arithmetical offsets) an absolute suffix -1, -2, ... related to the rank of the FactType.
   Affects only those events whose SortDates correspond to the Fact Date, as computed
   by a Date encoding algorithm. The algorithm does not handle Date modifiers so not all
   Event dates are handled, e.g. "Bef 1960".
*/
DROP TABLE IF EXISTS TmpFactOrder
;
CREATE TEMP TABLE IF NOT EXISTS TmpFactOrder
(Rank INTEGER PRIMARY KEY, FactName TEXT)
;
 
/* list of Fact Names, standard and custom, to be sorted, in rank order.
   Revise the list to suit your needs */
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Birth');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Christen');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Baptism');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Death');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Funeral');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Cremation');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Burial');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Obituary');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Memorial');
/* revise SortDates */
UPDATE EventTable
SET SortDate = SortDate
  -6692012023  -- this offset goes to Date-1 if the event is a ranked event
  *( (
      SELECT Rank
      FROM TmpFactOrder,
           FactTypeTable
      WHERE FactName LIKE Name
      AND FactTypeID = EventType
      )>0
    )
  +1048576  -- this offset adds steps of 1 to Date-1 multiplied by (rank-1)
  *( (
      SELECT Rank
      FROM TmpFactOrder,
           FactTypeTable
      WHERE FactName LIKE Name
      AND FactTypeID = EventType
      )-1
    ) -- maps the FactType to its order
WHERE EventID
IN (SELECT EventID FROM EventTable
    INNER JOIN
    (SELECT -- matching dates
     SortDate, OwnerID, COUNT()-1 AS Matches
     FROM EventTable
     INNER JOIN FactTypeTable
     ON EventType = FactTypeID
     WHERE EventTable.OwnerType = 0
     AND Name IN (SELECT FactName FROM TmpFactOrder)
     AND
     SortDate =   -- equals encoded event Date (if not a match, suggests that user has modified SortDate so don't touch it)
       (CASE
        WHEN DATE LIKE '.%'
        THEN 1
        ELSE Substr(DATE,3,5) END
        +10000
        )*562949953421312
        + Substr(DATE,8,2)*35184372088832
        + Substr(DATE,10,2)*549755813888
        + 17178820620
      GROUP BY SortDate, OwnerID, EventTable.OwnerType
     )
     USING (OwnerID, SortDate)
     INNER JOIN FactTypeTable
     ON EventType = FactTypeID
     WHERE Matches
     AND EventTable.OwnerType = 0
     AND Name IN (SELECT FactName FROM TmpFactOrder)
    )
;

Discussions & comments from Wikispaces site


chuckfilteau

Alternate method using the TmpFactOrder table

chuckfilteau
05 November 2016 18:50:42

Thanks for the algorithm and SQL code to set the SortDate.
From it, I created the TmpFactOrder table, but I added two additional columns (both integers)
FT – for the FactType that matched the FactName
ADJ – to pre-calculate each adjustment for each Rank
I then coded two simple update statement to update this table

◦Rationale – do the work only once, in this table, rather than redoing the work in SQL for each fact that had duplicate names **
UPDATE TmpFactOrder SET FT = (Select FactTypeID from FactTypeTable where FactName = Name);
UPDATE TmpFactOrder SET ADJ = ((Rank-1)*1048576) – 6692012023;

Finally, I came up with a much simpler, easier to understand, SQL statement that updated the SortDate for all events that had duplicate dates (for standard single date values)
Here is that statement:
UPDATE EventTable
SET SortDate=
((Substr(DATE,3,5) +10000)*562949953421312 — year
+ Substr(DATE,8,2)*35184372088832 — month
+ Substr(DATE,10,2)*549755813888 — day
+ 17178820620 — constant (time?)
+ (Select ADJ from TmpFactOrder — Rank adjustment (-1 to -20)
where EventType = FT)
)
where EventID in
(Select distinct E1.EventID — get the list of IDs that need to be changed
from EventTable as E1
JOIN EventTable as E2
Using (OwnerID,Date)
where E1.OwnerType=0
AND E1.[Date] like “D.%” — only events that have a single standard date
AND E1.EventType IN (Select FT from TmpFactOrder)
AND E1.EventID <> E2.EventID — DISCARD MATCHES ON THE SAME ROW
)
;
/* ALL DONE */


thejerrybryan

Minor “problem” with the handling of sort dates

thejerrybryan
11 September 2017 15:35:28

I am playing around with Tom’s original SortDate code rather than with chuckfilteau’s SortDate code. The code is very tricky (to me, at least), and the “problem” I have identified is most likely in myself rather than in Tom’s code.

I am using SQLite to add a Parents fact to each person who has parents. Having done so, I want the sort dates set such that the sort date for the Birth fact becomes birthdate_fact_date-1 and the sort date for the Parents fact becomes parents_fact_date-2, where birthdate_fact_date and parents_fact_date are the same date. So in addition to requiring a person actually to have parents before adding a Parents fact, my SQLite code also checks for the presence of a Birth fact and only adds the Parents fact if there is a Birth fact. For the newly added Parents fact, my code sets the Date, SortDate, and EditDate to be the same as for the Birth fact with other Parents fact fields being set appropriately for the Parents fact.

Having run my code to add the Parents fact, I copy Tom’s SortDate code, add my Parents fact to the list of facts, and run my copy of Tom’s code. All is mostly well, but all is not completely well. For an example when all is well, if the original birth date and sortdate is 18 May 1819, then my code sets the Parents fact date and sortdate to 18 May 1819. Tom’s code sets the Birth fact sortdate to 18 May 1819-1 and the Parents fact sortdate to 18 May 1819-2. Perfect.

For an example of when all is not well, I have a person with a birth date of 1790 and a sort date of 1 Jul 1790. So my newly added Parents fact also gets a date of 1790 and a sort date of 1 July 1790. Under these conditions, Tom’s code does not adjust the sort dates to 1 Jul 1790-1 and 1 Jul 1790-2, respectively. It thinks I have already set sort dates manually that should not be overridden. If before running Tom’s code, I change both sort dates to 1790, then Tom’s code changes the sort dates to 1790-1 and 1790-2, respectively. Perfect, except that all the sort dates in my database seem to be 1 Jul 1790 for year dates and something like 15 Mar 1790 for month dates like Mar 1790. And curiously, when I add a new year only date by hand, RM sets the sort date just to the year and not to 1 Jul of the year.

For another example of when all is not well, I have person with a birth date of Abt 1725 and a sort date of 1 Jul 1725. My code adds a Parents fact with a date of Abt 1725 and a sort date of 1 Jul 1725. Again, Tom’s code does not adjust the respective sort dates. I may not be understanding the code correctly, but I think the problem is not with the “Abt” modifier but again it’s because the code sees both events as having a 1 Jul 1725 sort date. If I manually change both sort dates to just plain 1725, then Tom’s code is not bothered by the “Abt” modifier and it does change the sort dates to 1725-1 and 1725-2, respectively. Again, this is perfect except that all my Abt dates for years have sort dates of 1 Jul for the appropriate year. I remember the use of 1 Jul sort dates being a standard RM technique for year only dates and my database is full of them, but now if I add a new year only fact by hand, the sort date is the year only without the 1 Jul. Has something changed in RM and I didn’t notice?

I thought I had come to a sufficient understanding of the sort date code to adjust it to meet my needs, but so far my efforts have failed. I can’t get it to override a Birth sort date of 1 Jul 1790 and a Parents sort date of 1 Jul 1790. I want the dates to become 1 Jul 1790-1 and 1 Jul 1790-2, respectively. Therefore, any advice would be most gratefully accepted.


thejerrybryan

thejerrybryan
11 September 2017 16:37:37

Aha (I think)!

I commented out the following code, and it seems to have solved my problem. I was trying to make changes within the commented out code, and I just needed to get rid of the commented out code entirely.

/* AND
SortDate = — equals encoded event Date (if not a match, suggests that user has modified SortDate so don’t touch it)
(CASE
WHEN DATE LIKE ‘.%’
THEN 1
ELSE Substr(DATE,3,5) END
+10000
)*562949953421312
+ Substr(DATE,8,2)*35184372088832
+ Substr(DATE,10,2)*549755813888
+ 17178820620 */

I would be interested in opinions concerning the following observation. It seems to me that commenting out the code in question might be a good idea for the general use case, not just for my specific use case. Which is to say that even with this code removed from the script, the script still does not modify sort dates that are different. For example, if the RM user has established a birth sort date of 1 Jul 1850-1 and a death sort date of 1 Jul 1850-2, then the script will not modify the sort dates even with the code commented out as indicated. The sort dates are not identical and are therefore are prevented from matching by INNER JOIN’s. They don’t need to be prevented from matching by the code that has been commented out. At least that’s the way it seems to me, and that’s what I’m going to go with for now.