Spouse Order – Global Sort by Marriage or First-Birth SortDate #spouse

When the person highlighted in Pedigree View or Family View is linked to more than one spouse, RootsMagic 7 lets you select the one to be shown from the dropdown Spouse list. This list also offers a “Rearrange spouses” control which sets the order of the spouses in the dropdown list and the order in which spouses are displayed in the Descendant and Timeline Views, the Edit Person screen and in certain reports. The Rearrange Spouses dialog allows both manual ordering and “Sort by Marriage Date”. This control operates only on the spouses of one person at a time. A wish has been expressed for a global Sort by Marriage Date. Furthermore, an emigre from Family Tree Maker pointed out that it also factors in the birth date of the first child in the case of an unwed couple. These requests may never see the light of day within RootsMagic but here is a script that does the global job in the interim.

Some words of caution. For spouses without marriage events or children, neither the script nor the current RootsMagic function can do anything but sort them to the top of the list. And this script will override sequences arranged manually through RM’s Rearrange Spouses, if there are two or more of the spouses with either a marriage event or child birth event.

Developed using SQLiteSpy and should require no special extensions so it should run with any recent SQLite manager.

SpouseSort2.sql
2016-01-12 Superseded SpouseSort.sql to factor in birth/christen date of first-born children of unwed couples in the absence of a marriage event.

As always, test it on a copy of your database or make a backup first.

Roles-Sort Alphabetically #roles #facttypes

RootsMagic 4-7 (current) list the user-added roles in the Edit Fact Type dialog in the order that they were created, after the builtin roles such as Principal and Witness. For fact types to which many roles have been added, it is desirable that they be sorted alphabetically. While this should easily be done within the program for display purposes and has been requested repeatedly over years, until such an enhancement is provided here is a script that uses a brute force method to rearrange the rows in the data table so that the role names are presented alphabetically.
RoleSortBeforeAfter.png
As far as I can tell, the script is harmless. There is no need to use RootsMagic’s Rebuild Indexes after its execution but it would be advisable to test it on a copy of your database or make a backup before applying it. It was developed using SQLiteSpy but uses no esoteric functions so it should run with any SQLite manager that supports a fake RMNOCASE collation extension..

Caveat: If a future version of RM increases the number of builtin roles to more than 58, every instance of 58 in the script must be revised accordingly else those builtin roles above the RoleID of 58 will be mixed up with a user-added role after a drag’n’drop or GEDCOM transfer to another RootsMagic database..

Tip: If you want the role names to appear in an order sorted by your preference, rename them prefixed by a number, e.g.:

  1. with Parents
  2. with Father
  3. with Grandparents
  4. Other

If you have more than 9 roles, then use double digits:
01 with Parents
02 with Father

RoleSort.sql

Group – Persons with Duplicate Events #namedgroup #events #date #duplicates

2023-02-19 Version for #RM8 added.

  • Creates a Named Group in the open database with the name “SQL: Duplicate Events” if one does not already exist
  • Removes all members from that named group
  • Populates that named group with persons having “duplicate” events

Duplication is declared if a pair of events:

  • (have dates for which the span of calendar years (ignoring month and day) is less than a parameter (2 in the script) OR
  • do not have dates but are fact types that use dates) AND (are both set or both not set as Primary events)

Duplication is resolved in RM by:

  • deleting an event (you may want to incorporate an alternative date in the date or notes of the remaining event) OR
  • setting one event only of the pair as Primary

To refresh the named group’s membership list to reflect corrections you may have made to resolve duplication of events, rerun the query to regenerate the RootsMagic group table and then, in RootsMagic, select a different group and then re-select the “SQL: Duplicate Events” group. You should see that names have disappeared from the group, provided you have corrected all the duplicate events for each person you worked on. The Quick Groups feature in RM7 and the Groups list in RM8 allow you to unmark a member from the group right from the Edit Person screen making it unnecessary to rerun the script unless you missed doing so.

To vary the parameter controlling how sensitive the query is to date differences, change this line at the end of the script:

WHERE ABS(SUBSTR(Date1,4,4) - SUBSTR(Date2,4,4)) <2 -- this controls how tight the year match must be.
-- in future, the date comparison could call on a date calculator that could compute the difference in days
 
)
; --END of SCRIPT

A setting of “<2” permits two consecutive years which allows Dec 31, 1900 and Jan 1, 1901 to be considered duplicates as it does for Jan 1, 1900 and Dec 31, 1901. A setting “<1” requires the events to be in the same year.

Group-PersonsWithDuplicateEvents.sql rev 2018-07-07 bug fixed when no other existing group

RM8 Version 2023-02-19

Nickname Manipulations #nicknames

Why?

RootsMagic has three different storage mechanisms for Nicknames:

  1. Nickname field in each Name record
  2. In quotation marks or parentheses in the Given field of each Name record
  3. As above in a Name record of type “Nickname”

Different outputs from RootsMagic have differing usage of these. Box charts cannot use Alternate Names and only include the primary name Nickname field if Prefix and Suffix are also included. Sentence templates for facts other than the Alternate Name type cannot access Alternate Names, cannot parse the nickname embedded in the primary Given field but can use the primary Nickname field. So there are outputs where it may be desirable that nicknames be embedded with the given name and others where it should not be. And as a database is built over time from a variety of sources, there may be inconsistency in the way nicknames are stored within the database.

This page offers a set of scripts that can manipulate where the nickname values are stored, globally and quickly, to optimise for a desired output and to make consistent.

Combine | Split Nickname and Given Names

NickGivenSplitCombine.png
These two scripts can swap the nickname between the NickName field and the Given Names field. The splitter works only with quotation marks delimiting the nickname in the Givens field; the combiner puts the nickname in quotes in the Givens field. You can use RootsMagic Search and Replace function on the Given names to convert all instances of parentheses to double quotation marks.
NickGivenSplit.sql
NickGivenCombine.sql

Ambivalent Nicknames

This script returns Record Number and Names of Persons who have both an embedded Nickname and a value in the Nickname field. These cannot be processed by NickGivenCombine.sql and NickGivenSplit.sql and should be resolved in the database with one or the other but not both.

Includes those having parentheses to denote nickname in the Given field instead of quotes; parentheses not supported by other scripts in this series.

NicknameAmbivalent.sql

Conflict between Primary and Alternate Nicknames

This script returns the Record Number and Nicknames of Persons who have both an Alternate Nickname and a Primary name with an embedded nickname or non-empty NickName field and the nickname values in the corresponding fields do not match. Use RIN to find the person in RootsMagic Explorer to revise as needed.
NickAltNameConflict.sql

UNDER CONSTRUCTION
-possibly converters between the nickname fields of Primary Name and Alternate Name of type Nickname but this would be complex.

Media Users List Query #media #rm8

Update

As of RootsMagic 8 Preview 7.9.180, the traceability of where an image is used has changed little from what was described below for RM4. The need for an outboard query such as this is as high in early 2021 as it was in 2010. With some changes in tables from RM7, the Media-UsersList4.sql query does not run on RM8. Pat Jones revised it and applied a technique she describes at Common Table Expressions – The Building Blocks of SQL to streamline it:

Media-UsersList4-RM8.sql

For a version that works with databases from RM5 to RM7:
Media-UsersList4-RM7.sql 

Introduction

RootsMagic 4 falls short in handling media in several ways. One serious one is its inability to report fully all the ‘owners’ of the media, i.e., its terminology to describe what person, couple, event, source, citation, place or place detail to which a media item is attached, in sufficient detail that the attachment can be found. Trace-ability is essential to be able to make changes to an attachment’s properties, i.e., its caption, description, reference number, date, sort date, primary and scrapbook inclusion flags. RM4’s Report > Lists > Multimedia List does not even report usage by Sources or Source Details and fails to report the Place to which a Place Detail, having media attached, belongs. Similarly, the Media Gallery > Tools > “Show where this media is used” results are incomplete for media attached to Source Details or to Place Details, giving little or no clue to how you would navigate to the citation or Place where you could then drill down to the Media and edit the caption or other properties.

Here’s an example of what I mean:

Owner according to Media Gallery > Tools > “Show where this media is used”

Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete) (citation)

How can one possibly find the citation without at least the RIN or name of the person?

Users according to Media-UsersList query

(citation) WICKENS, Martha Jane "Jenny"-208 : RESIDENCE (FAMILY) ca 1891 : HOLDEN, Matthew Edwin-149 citing Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete)
(citation) HOLDEN, Matthew Edwin-149 : RESIDENCE (FAMILY) ca 1891 : WICKENS, Martha Jane "Jenny"-208 citing Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete)

Both RM4 and the query show that the media file is attached to a citation but the new query provides explicit guidance to the citation via not only the name and RIN of the person but also the fact name and date so that the citing fact can be quickly found and its Citation Manager opened. The source name is shown (that’s all the Media Gallery shows apart from the caption and full filename) so it can be readily selected from the Citation Manager and the Source Detail Media Album opened therefrom and the media item selected by file name for the editing of its properties. That is the only way RM4 lets you get to the item’s properties; obviously, a road map is necessary. This query shows the route.

Download

Media-UsersList.sql – original query, easier to modify, fast on smaller databases, bogs down on large ones
Media-UsersList3.sql RMtrix_tiny_check.png – fast even on very large databases, larger sql file, harder to modify at the base SELECTs.
Media-UsersList4.sql – as ver 3 but revised for RM5 and 6; prior versions work with RM4.

Screenshots

With Thumbnail Images

Using SQLite Expert Personal, make sure the following setting is checked for the thumbnail image to be shown:

Tools > Options > Data:

  • Results-Display results in grid
  • Grids-Show images
Media-UsersList_Screenshot_from_Expert.PNG
SQLite Expert Personal can display the thumbnail images stored in the database. A subset of the available columns is displayed to reduce clutter. Unfortunately, column widths are not remembered on re-executing the same query. Also, sorting on column heading does not work unless the ORDER BY clause at the very end of the query is deleted.

Without Thumbnail Images

Media-UsersList_Screenshot_from_Spy.PNG
Screenshot from SQLiteSpy which has no image display capability but does remember column widths within the same session and supports sorting on column regardless of an ORDER BY in the query.

Discussions & comments from Wikispaces site


ve3meo

 

Comment: “Download”

ve3meo
03 September 2018 21:38:03

ve3meo Nov 14, 2011

This query can bog down on very large databases because it produces a very large set of intermediate results that must be stored in memory. It is being (slowly) revised to produce many small sets of intermediate results so that it will be able to handle large databases efficiently. This is not a trivial re-write.
ve3meo Nov 14, 2011

Re-write done: Media-UsersList3.sql. On a large database that was taking 37 to 120 seconds to display results, the time has dropped to 1-5s, depending on the SQLite manager, what else is going on, how much memory is available.The key was to structure it to minimise the size of intermediate results that are then JOINed with another result set.

Citations Invisible – Reveal and Revert #citations #sources #ftm2014 #ftm2012 #ancestrycom #gedcom

With the flurry of users of Family Tree Maker looking to migrate their database to RootsMagic, an outcry has gained volume because of the disappearance of some citations. This problem was described here in 2012 in Citations Invisible Revealed which offered two scripts, both of which are available in the RMtrix utility, one to find such citations for the Primary (Preferred Name), the other to convert them to citations for the Person. That was an irreversible conversion. In the hope that RootsMagic will eventually support citations for the Primary (Preferred) Name, this pair of scripts provides both conversion and reversion. As of 2015-12-24, RootsMagic is expected to issue an update that will do a similar conversion and an upgrade that will support citations for the Primary (Preferred) Name and revert these conversions. However, it will use a different value in the flag to make the citations reversible than was used in the initial scripts, necessitating these revised versions to be compatible.

Citations, Invisible – Convert to Personals + Flagged.sql rev 2015-12-24 Flags field now set bitwise on 1st bit instead of to 7
Citations, Invisible Revealed – Revert – Bitwise.sql 2015-12-24 Unsets 1st bit of Flags field instead of setting to 0

Caveat! To make it reversible, the converter uses the apparently unused Flags column in the CitationTable. The initial converter dated 2015-12-19 set this value to 7, corresponding to the original OwnerType value for the Citation Table record; the value that RootsMagic will use for this purpose will be the 1st bit set to 1, allowing the Flags field to be used for many bitwise flags. I cannot predict what the consequences might be for those records having a flag value of 7 set by the initial converter. If you used the now-deprecated, initial version of Citations, Invisible – Convert to Personals + Flagged.sql, it would be advisable to run this complementary script to revert them to the primary Name fact: Citations, Invisible Revealed – Revert.sql Then follow up with the revised converter above or wait for the RootsMagic update to do the same thing.

James Clouse posted this video to the RootsMagic Users Facebook Group showing how to use SQLiteSpy to open your database and run the conversion script. Click on the settings and select HD to avoid fuzziness.

Citations Invisible Revealed #citations #ancestrycom

N.B. This is now the background page for a pair of revised scripts that supersede the conversion script here and make the process reversible. See Citations Invisible – Reveal and Revert.

First reported by Paul Harris in the RootsMagic Forums in the discussion Unused Citations after Import, we have concluded that there is a bug in RootsMagic 5.0.4.1 and, possibly, earlier versions that suppresses or fails to include a certain type of citation from appearing in key user screens and reports. Yet these citations are generated by RootsMagic, can be seen in the Source List report and survive a drag and drop transfer to a new database.

Citations, Invisible – List.sql RMtrix_tiny_check.png This query will show whether there are any such ‘invisible’ citations in a RootsMagic database.
Citations, Invisible – Convert to Personals.sql RMtrix_tiny_check.png This query will convert these ‘invisible’ citations to a type that is visible in the RootsMagic program.

The ‘invisible’ citations covered by these queries are for the Primary Name of a person and are converted to a Personal citation to become visible. We think an Ancestry citation that “provides evidence for Name” is exported as a citation for the “Preferred” Name, when there are Alternate Names, not for the Person. If there are no Alternate Names, we surmise that Ancestry exports the citation against the Person (top level in GEDCOM, Name is lower); these are visible. Citations for Alternate Names are visible and correctly placed in the Edit Person screen. Since there is no other place for a Primary Name citation to go than beside the name at the top of the Edit Person list and since that is where Personal citations do get flagged, then it is reasonable to convert the ‘invisible’ citations to Personal ones.

On import, RootsMagic assigns an OwnerType = 7 in the CitationTable for citations for Names, be they Preferred or Alternate, signifying that the corresponding CitationTable.OwnerID points to a unique NameID in the NameTable. Each unique NameID has a set of names for a Person in the PersonTable whose unique PersonID corresponds to the NameTable.OwnerID. For a given NameTable.OwnerID (or Person), there should be but one record with the IsPrimary column set to 1 – this is the Primary Name; all other records in the set have IsPrimary = 0, signifying they are Alternate Names. The List query lists all citations of Type 7 that point to a Primary Name.

For Personal citations, OwnerType = 0 in the CitationTable, signalling to RootsMagic to match the CitationTable.OwnerID with PersonTable.PersonID and that this is a Personal citation.

The Convert to Personals query replaces the CitationTable.OwnerID with the PersonID owning the NameID pointed to by the citation and sets OwnerType to 0 for only those citations for a Primary Name (OwnerType = 7, linked NameTable.IsPrimary = 1 or True).

Dates – Same Day Sort Order #date #sortdate #update

#RM10 compatible

Problems

Some RootsMagic users bemoan the fact that events having the same date are displayed and reported in an unnatural or illogical order, having more to do with the order they were entered. Burial before Death or Christen before Birth are often mentioned. RootsMagic does provide the user with the option to modify the SortDate date to establish the desired order.

RM’s proprietary SortDate is lost in an export to another program; any import into RM from anything other than a RM database or an RM generated GEDCOM with the Extra Details (RM specific) box checked will be assigned SortDates matching the event date resulting in the probable loss of the desired order. So manual control over the SortDate is of no comfort to those who move their database between different programs to take advantage of their respective strengths. An enhancement to provide better results for the order of same day events has long been on the RootsMagic Wish List.

Workaround

Meanwhile, here are two SQLite queries that can re-order all same day events of limited types in seconds with results as shown in the screenshots below. They both exploit the SortDate feature that values of the form date-n, where n is an integer from 1 to 4999, are sorted in ascending order.

Dates-SameDaySortOrder.png
Before and after the query was applied.

SortDateSameDayOrder.sql

SortDateSameDayOrder.sql, currently addresses the following in the order listed: Birth, Christen, Baptism, Death, Cremation and Burial. Others could be added. Should work with any SQLite manager on both RM4 and RM5 databases. Currently deprecated because it changes other facts on the same day; left here because its comparative simplicity may help to understand its successor below.

/* SortDateSameDayOrder.sql
   2011-12-19 ve3meo
   Alters SortDates of Birth, Christen, Baptism, Death, Cremation and Burial
   to a natural order when any pair or more occur on the same date.
   Could be extended to order other facts also. SortDates are effectively assigned
   (by arithmetical offsets) an absolute suffix -1, -2, ... related to the FactType.
   Affects only those events whose SortDates correspond to the Fact Date, as computed
   by a Date encoding algorithm. The algorithm does not handle Date modifiers so not all
   Event dates are handled, e.g. "Bef 1960".
   2011-12-20 order of Cremation and Burial corrected
*/
 
UPDATE EventTable
SET SortDate = SortDate-6692012023*(EventType
  IN (1,3,7,2,4,5))  -- list of FactTypes we want to sort, in no particular order except this corresponds to the desired order
  +1048576*(SUBSTR('1426503',EventType,1)-1) -- the substr maps the FactType to its order
WHERE EventID
IN (SELECT EventID FROM EventTable
    INNER JOIN
    (SELECT -- matching dates
     SortDate, OwnerID, COUNT()-1 AS Matches FROM EventTable
     WHERE OwnerType = 0 AND EventType IN (1,3,7,2,5,4)
     AND
     SortDate =   -- equals encoded event Date (if not a match, suggests that user has modified SortDate so don't touch it)
       (CASE
        WHEN DATE LIKE '.%'
        THEN 1
        ELSE Substr(DATE,3,5) END
        +10000
        )*562949953421312
        + Substr(DATE,8,2)*35184372088832
        + Substr(DATE,10,2)*549755813888
        + 17178820620
      GROUP BY SortDate, OwnerID, OwnerType
     )
     USING (OwnerID, SortDate)
     WHERE Matches
    )
;

SortDateSameDayOrderCustom.sql

SortDateSameDayOrderCustom.sql RMtrix_tiny_check.png
Goes to the next level, supporting the ranking of any number of facts, be they standard or custom. It also corrects an oversight in SortDateSameDayOrder.sql and does not touch unranked facts on the same date. Requires the use of a fake RMNOCASE collation or possible revision with COLLATE NOCASE phrases where FactTypeTable.Name is used. Because SortDate supports up to date-4999, this query could be modified to have a multiplier of the rank, thus leaving room for other events on the same date to be manually added with a SortDate lying between those assigned by the query, without having to revise the existing SortDates.

/* SortDateSameDayOrderCustom.sql
   2011-12-20 ve3meo
   Alters SortDates of any set of Fact types
   to a natural order when any pair or more occur on the same date.
   Could be extended to order other facts also. SortDates are effectively assigned
   (by arithmetical offsets) an absolute suffix -1, -2, ... related to the rank of the FactType.
   Affects only those events whose SortDates correspond to the Fact Date, as computed
   by a Date encoding algorithm. The algorithm does not handle Date modifiers so not all
   Event dates are handled, e.g. "Bef 1960".
*/
DROP TABLE IF EXISTS TmpFactOrder
;
CREATE TEMP TABLE IF NOT EXISTS TmpFactOrder
(Rank INTEGER PRIMARY KEY, FactName TEXT)
;
 
/* list of Fact Names, standard and custom, to be sorted, in rank order.
   Revise the list to suit your needs */
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Birth');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Christen');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Baptism');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Death');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Funeral');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Cremation');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Burial');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Obituary');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Memorial');
/* revise SortDates */
UPDATE EventTable
SET SortDate = SortDate
  -6692012023  -- this offset goes to Date-1 if the event is a ranked event
  *( (
      SELECT Rank
      FROM TmpFactOrder,
           FactTypeTable
      WHERE FactName LIKE Name
      AND FactTypeID = EventType
      )>0
    )
  +1048576  -- this offset adds steps of 1 to Date-1 multiplied by (rank-1)
  *( (
      SELECT Rank
      FROM TmpFactOrder,
           FactTypeTable
      WHERE FactName LIKE Name
      AND FactTypeID = EventType
      )-1
    ) -- maps the FactType to its order
WHERE EventID
IN (SELECT EventID FROM EventTable
    INNER JOIN
    (SELECT -- matching dates
     SortDate, OwnerID, COUNT()-1 AS Matches
     FROM EventTable
     INNER JOIN FactTypeTable
     ON EventType = FactTypeID
     WHERE EventTable.OwnerType = 0
     AND Name IN (SELECT FactName FROM TmpFactOrder)
     AND
     SortDate =   -- equals encoded event Date (if not a match, suggests that user has modified SortDate so don't touch it)
       (CASE
        WHEN DATE LIKE '.%'
        THEN 1
        ELSE Substr(DATE,3,5) END
        +10000
        )*562949953421312
        + Substr(DATE,8,2)*35184372088832
        + Substr(DATE,10,2)*549755813888
        + 17178820620
      GROUP BY SortDate, OwnerID, EventTable.OwnerType
     )
     USING (OwnerID, SortDate)
     INNER JOIN FactTypeTable
     ON EventType = FactTypeID
     WHERE Matches
     AND EventTable.OwnerType = 0
     AND Name IN (SELECT FactName FROM TmpFactOrder)
    )
;

Discussions & comments from Wikispaces site


chuckfilteau

Alternate method using the TmpFactOrder table

chuckfilteau
05 November 2016 18:50:42

Thanks for the algorithm and SQL code to set the SortDate.
From it, I created the TmpFactOrder table, but I added two additional columns (both integers)
FT – for the FactType that matched the FactName
ADJ – to pre-calculate each adjustment for each Rank
I then coded two simple update statement to update this table

◦Rationale – do the work only once, in this table, rather than redoing the work in SQL for each fact that had duplicate names **
UPDATE TmpFactOrder SET FT = (Select FactTypeID from FactTypeTable where FactName = Name);
UPDATE TmpFactOrder SET ADJ = ((Rank-1)*1048576) – 6692012023;

Finally, I came up with a much simpler, easier to understand, SQL statement that updated the SortDate for all events that had duplicate dates (for standard single date values)
Here is that statement:
UPDATE EventTable
SET SortDate=
((Substr(DATE,3,5) +10000)*562949953421312 — year
+ Substr(DATE,8,2)*35184372088832 — month
+ Substr(DATE,10,2)*549755813888 — day
+ 17178820620 — constant (time?)
+ (Select ADJ from TmpFactOrder — Rank adjustment (-1 to -20)
where EventType = FT)
)
where EventID in
(Select distinct E1.EventID — get the list of IDs that need to be changed
from EventTable as E1
JOIN EventTable as E2
Using (OwnerID,Date)
where E1.OwnerType=0
AND E1.[Date] like “D.%” — only events that have a single standard date
AND E1.EventType IN (Select FT from TmpFactOrder)
AND E1.EventID <> E2.EventID — DISCARD MATCHES ON THE SAME ROW
)
;
/* ALL DONE */


thejerrybryan

Minor “problem” with the handling of sort dates

thejerrybryan
11 September 2017 15:35:28

I am playing around with Tom’s original SortDate code rather than with chuckfilteau’s SortDate code. The code is very tricky (to me, at least), and the “problem” I have identified is most likely in myself rather than in Tom’s code.

I am using SQLite to add a Parents fact to each person who has parents. Having done so, I want the sort dates set such that the sort date for the Birth fact becomes birthdate_fact_date-1 and the sort date for the Parents fact becomes parents_fact_date-2, where birthdate_fact_date and parents_fact_date are the same date. So in addition to requiring a person actually to have parents before adding a Parents fact, my SQLite code also checks for the presence of a Birth fact and only adds the Parents fact if there is a Birth fact. For the newly added Parents fact, my code sets the Date, SortDate, and EditDate to be the same as for the Birth fact with other Parents fact fields being set appropriately for the Parents fact.

Having run my code to add the Parents fact, I copy Tom’s SortDate code, add my Parents fact to the list of facts, and run my copy of Tom’s code. All is mostly well, but all is not completely well. For an example when all is well, if the original birth date and sortdate is 18 May 1819, then my code sets the Parents fact date and sortdate to 18 May 1819. Tom’s code sets the Birth fact sortdate to 18 May 1819-1 and the Parents fact sortdate to 18 May 1819-2. Perfect.

For an example of when all is not well, I have a person with a birth date of 1790 and a sort date of 1 Jul 1790. So my newly added Parents fact also gets a date of 1790 and a sort date of 1 July 1790. Under these conditions, Tom’s code does not adjust the sort dates to 1 Jul 1790-1 and 1 Jul 1790-2, respectively. It thinks I have already set sort dates manually that should not be overridden. If before running Tom’s code, I change both sort dates to 1790, then Tom’s code changes the sort dates to 1790-1 and 1790-2, respectively. Perfect, except that all the sort dates in my database seem to be 1 Jul 1790 for year dates and something like 15 Mar 1790 for month dates like Mar 1790. And curiously, when I add a new year only date by hand, RM sets the sort date just to the year and not to 1 Jul of the year.

For another example of when all is not well, I have person with a birth date of Abt 1725 and a sort date of 1 Jul 1725. My code adds a Parents fact with a date of Abt 1725 and a sort date of 1 Jul 1725. Again, Tom’s code does not adjust the respective sort dates. I may not be understanding the code correctly, but I think the problem is not with the “Abt” modifier but again it’s because the code sees both events as having a 1 Jul 1725 sort date. If I manually change both sort dates to just plain 1725, then Tom’s code is not bothered by the “Abt” modifier and it does change the sort dates to 1725-1 and 1725-2, respectively. Again, this is perfect except that all my Abt dates for years have sort dates of 1 Jul for the appropriate year. I remember the use of 1 Jul sort dates being a standard RM technique for year only dates and my database is full of them, but now if I add a new year only fact by hand, the sort date is the year only without the 1 Jul. Has something changed in RM and I didn’t notice?

I thought I had come to a sufficient understanding of the sort date code to adjust it to meet my needs, but so far my efforts have failed. I can’t get it to override a Birth sort date of 1 Jul 1790 and a Parents sort date of 1 Jul 1790. I want the dates to become 1 Jul 1790-1 and 1 Jul 1790-2, respectively. Therefore, any advice would be most gratefully accepted.


thejerrybryan

thejerrybryan
11 September 2017 16:37:37

Aha (I think)!

I commented out the following code, and it seems to have solved my problem. I was trying to make changes within the commented out code, and I just needed to get rid of the commented out code entirely.

/* AND
SortDate = — equals encoded event Date (if not a match, suggests that user has modified SortDate so don’t touch it)
(CASE
WHEN DATE LIKE ‘.%’
THEN 1
ELSE Substr(DATE,3,5) END
+10000
)*562949953421312
+ Substr(DATE,8,2)*35184372088832
+ Substr(DATE,10,2)*549755813888
+ 17178820620 */

I would be interested in opinions concerning the following observation. It seems to me that commenting out the code in question might be a good idea for the general use case, not just for my specific use case. Which is to say that even with this code removed from the script, the script still does not modify sort dates that are different. For example, if the RM user has established a birth sort date of 1 Jul 1850-1 and a death sort date of 1 Jul 1850-2, then the script will not modify the sort dates even with the code commented out as indicated. The sort dates are not identical and are therefore are prevented from matching by INNER JOIN’s. They don’t need to be prevented from matching by the code that has been commented out. At least that’s the way it seems to me, and that’s what I’m going to go with for now.

Recursive Queries – Ancestors and Descendants #recursive #ancestors #descendants

As of Feb 3, 2014 and version 3.8.3, SQLite introduced “Recursive common table expressions (that) provide the ability to do hierarchical or recursive queries of trees and graphs, a capability that is not otherwise available in the SQL language”. (http://www.sqlite.org/lang_with.html) This means it is now possible to generate ancestry or descendancy lists within SQLite, without having to resort to a higher level language. These lists can be unlimited in length, unlike queries based on earlier versions of SQLite, which required a SELECT statement for each generation (e.g. the 2011 Ancestors Query). The WITH RECURSIVE syntax is much shorter than cascading like queries and executes much faster (60ms vs 300ms, for example). While the syntax is challenging to understand, an error or change needs to be dealt with in one or two spots, not in every SELECT in a cascade. Here’s a helpful guide to gain understanding: The Simplest SQLite Common Table Expression Tutorial.

A first attempt at a recursive query was reported on Ancestors Query – Recursive. I then proceeded to develop the complement – a recursive Descendants query. Because the two are complementary and will benefit from parallel development, and because recursion is such a significant step forward in the power of SQLite, a page focussed on recursion as it applies to the two essential genealogical queries seems appropriate. This page presents an enhanced Ancestors query and its complementary Descendants query and will, hopefully, see further development.

rev 2015-11-24 Descendants: filter added for sex so that a birth only relationship for female descendants of a woman would be those sharing mtDNA from her; the complementary male line is also provided which would those sharing the Y-STR chromosome. Note that RootsMagic has in its Select People dialogs tools to mark/unmark people by genetic lines.

Ancestors

This query is fundamentally the same as Ancestors Query – Recursive with the addition of named parameters which the user defines at run-time:

  1. $BirthOnly(YN): enter “Y” or “y” to restrict the lineage to birth relationships (bloodlines); entry of anything else will include all relationships (adoptive, step, etc.)
  2. $Person(RIN): enter the record number of the person whose list of ancestors is wanted

The result is a list of the record numbers of the ancestors under the heading AncestorID and does not include the starting RIN.
AncestorsRecursive.sql

-- AncestorsRecursive.sql
/*
2014-04-14 Tom Holden ve3meo
 
Generates the list of RINs for the ancestors of a person.
 
Requires support not only for SQLite 3.8.3 or later but also
for named parameters for user input of the RIN of the starting person
and choice of birth only or all relationships.
 
Developed and tested with current SQLite Expert Personal 3.5.36.2456
 
Uses the WITH RECURSIVE syntax introduced in SQLite 3.8.3 2014-02-03
modelled on http://www.sqlite.org/lang_with.html example
and complement of DescendantsRecursive.sql
*/
 
WITH RECURSIVE
  parent_of(ChildID, ParentID) AS
    (SELECT PersonID, FatherID AS ParentID FROM PersonTable
       LEFT JOIN ChildTable ON PersonID=ChildTable.ChildID
       LEFT JOIN FamilyTable USING(FamilyID)
       WHERE
         CASE $BirthOnly(YN)
         WHEN 'Y' OR 'y' THEN RelFather=0
         ELSE 1
         END
         --RelFather=0 --birth father (ELSE WHERE 1 to include all relationships)
     UNION
     SELECT PersonID, MotherID AS ParentID FROM PersonTable
       LEFT JOIN ChildTable ON PersonID=ChildTable.ChildID
       LEFT JOIN FamilyTable USING(FamilyID)
       WHERE
         CASE $BirthOnly(YN)
         WHEN 'Y' OR 'y' THEN RelMother=0
         ELSE 1
         END
         --RelMother=0 --birth mother (ELSE WHERE 1 to include all relationships)
     ),
  ancestor_of_person(AncestorID) AS
    (SELECT ParentID FROM parent_of
       WHERE ChildID=$Person(RIN) --enter RIN of starting person at runtime
     UNION --ALL
     SELECT ParentID FROM parent_of
       INNER JOIN ancestor_of_person ON ChildID = AncestorID)
SELECT AncestorID FROM ancestor_of_person, PersonTable
 WHERE ancestor_of_person.AncestorID=PersonTable.PersonID
;

Descendants

This query is a complement to the Ancestors query, structurally very similar, and with the same named parameters which the user defines at run-time:

  1. $BirthOnly(YN): enter “Y” or “y” to restrict the lineage to birth relationships (bloodlines); entry of anything else will include all relationships (adoptive, step, etc.)
  2. $Sex(MF?): enter one of “MmFf) to restrict the lineage to males or females only; anything else includes both.
  3. $Person(RIN): enter the record number of the person whose list of descendants is wanted

The result is a list of the record numbers of the descendants under the heading DescendantID and does not include the starting RIN.
DescendantsRecursive.sql rev 2015-11-24 filter added for sex so that a birth only relationship for female descendants of a woman would be those sharing mtDNA from her. The code following is the original query.

-- DescendantsRecursive.sql
/*
2014-04-14 Tom Holden ve3meo
 
Generates the list of RINs for the descendants of a person.
 
Requires support not only for SQLite 3.8.3 or later but also
for named parameters for user input of the RIN of the starting person
and choice of birth only or all relationships.
 
Developed and tested with current SQLite Expert Personal 3.5.36.2456
 
Uses the WITH RECURSIVE syntax introduced in SQLite 3.8.3 2014-02-03
modelled on http://www.sqlite.org/lang_with.html example
and complement of AncestorsRecursive.sql
*/
 
WITH RECURSIVE
  child_of(ParentID, ChildID) AS
    (SELECT PersonID, ChildTable.ChildID FROM PersonTable
       LEFT JOIN FamilyTable ON PersonID=FatherID
       LEFT JOIN ChildTable USING(FamilyID)
       WHERE
         CASE $BirthOnly(YN)
         WHEN 'Y' OR 'y' THEN RelFather=0
         ELSE 1
         END
         --RelFather=0 --birth father (ELSE WHERE 1 to include all relationships)
     UNION
     SELECT PersonID, ChildTable.ChildID FROM PersonTable
       LEFT JOIN FamilyTable ON PersonID=MotherID
       LEFT JOIN ChildTable USING(FamilyID)
       WHERE
         CASE $BirthOnly(YN)
         WHEN 'Y' OR 'y' THEN RelMother=0
         ELSE 1
         END
         --RelMother=0 --birth mother (ELSE WHERE 1 to include all relationships)
     ),
  descendant_of_person(DescendantID) AS
    (SELECT ChildID FROM child_of
       WHERE ParentID=$Person(RIN) --enter RIN of starting person at runtime
     UNION --ALL
     SELECT ChildID FROM child_of
       INNER JOIN descendant_of_person ON ParentID = DescendantID)
SELECT DescendantID FROM descendant_of_person, PersonTable
 WHERE descendant_of_person.DescendantID=PersonTable.PersonID
;

To Do’s

  1. Update previous queries that used the old cascaded Ancestors query, e.g., Ancestors Named Group
  2. Create new queries based on these simple lists of ancestors and descendants, e.g. Named Groups and Color Coding
  3. Investigate how one might add collateral lines
  4. How might one apply the ancestry query to clearing the Living flag for ancestors of persons who should be dead.
  5. Add generation number to results of ancestors and descendants – useful for reports
  6. Add FamilyID to results

Source Templates – Versatile Free Form Hybrids #sourcetemplates #sources

Introduction

This query introduces a radical variation on RootsMagic’s system of Source Templates. It combines Source Templates and Free Form properties into Versatile Free Form Hybrid Source Templates incorporating the best of both worlds. Sources built on older source templates will now have free form fields that can be used to create alternative sentences that will survive GEDCOM export better than the templated sentences. Sources built on Free Form are not changed but can now be converted losslessly to templated sources, the data values being transferred from the Free Form source to the Free Form-like fields in the Hybrid source. And new sources built on the Hybrid templates can begin either in Free Form or template style while the other can be created and edited from within the common Edit Source window.

This idea sprang from discussion on Jerry Bryan’s page Question with respect to Changing a Master Source to a Different Source Template.

Conversion from ordinary templates to Versatile Free Form Hybrid Source Templates

Screenshots

SourceTemplatesHybridFreeFormConvert-EditSrc1.PNG
Source using a Hybrid built-in source template showing the default output sentences coming from the original template fields, not the added (FF) fields.
SourceTemplatesHybridFreeFormConvert-EditSrc2.PNG
Same source but the FORCE FF field non-empty value causes the sentence outputs to come from the (FF) fields.

Restoring after Drag ‘n’ Drop to new database

Transfer of sources, based on built-in templates that have been so Hybridized, to another RM database initially appears to lose the Free Form fields. However, the data values were transferred and merely running the query on the other database will restore the transferred sources to the Hybrid state.

Conversion of Free Form sources to Hybrids

Conversion of Free Form sources to Hybrid templates is possible and not difficult. You need to replace the TemplateID value of 0 for specific sources in the SourceTable to the TemplateID value of the desired Hybrid template. The (FF) fields will be populated by the values of the Free Form source. If that template does not work out, you can readily change the source to another Hybrid template and not lose the (FF) values from which you parse values for the template fields.

Feedback

This query is EXPERIMENTAL – try it on a copy of your database. See if it works for you. Let me know what you think, what works, what doesn’t. For those who have used !MyFreeForm3, I suspect there will be complications that will be precluded in a later version – after all, I use it, but this initial version was developed on just the built-in source templates. who

Future Considerations

A later version will add the fields from !MyFreeForm3 for even better Free Form style sentences and another utility could toggle the ForceFF field to control which sentence template is used for export, i.e., Free Form for export, templated for reports and other outputs.

Download SQL Script

SourceTemplatesHybridFreeFormConvert.sql rev 2013-12-17: !MyFreeForm3 templates no longer converted; speed tweaks.

Script

--SourceTemplatesHybridFreeFormConvert.sql
/*
2013-12-16 Tom Holden ve3meo
2013-12-17 rev1. no longer converts !MyFreeForm3 templates
           rev2. possible speed up by reducing use of CAST
Marries source templates up with Free Form for the best of both worlds:
a) use the templated part as an input form to help draft a Free Form source
b) paste a citation from Ancestry, FamilySearch et al into the Free Form part to
refer to while populating the template fields.
c) toggle between sentences outputted from either the Free Form fields or the
templated fields
 
EXPERIMENTAL
The revised templates have added fields in Free Form style (FF) and a Force FF field
that determines whether the sentences outputted are from the Free Form fields or
the template fields.
 
N.B.: the built-in source templates, while converted, will not transfer
to another database. Because the target database has only the templated fields
in the template definitions, the Free Form fields will appear to be lost. Running
this utility on the target database will make them reappear with the values in
each source intact.
 
*/
 
-- backup SourceTemplateTable builtin templates (you could revert to this version
-- after executing the query the first time)
 
DROP TABLE IF EXISTS xSourceTemplateTableBak
;
CREATE TABLE IF NOT EXISTS xSourceTemplateTableBak AS
SELECT * FROM SourceTemplateTable
;
 
/*
-- Restore from backup
INSERT OR REPLACE INTO SourceTemplateTable
SELECT * FROM xSourceTemplateTableBAK
*/
 
DROP TABLE IF EXISTS xTmpUnconvertedTemplates
;
CREATE TEMP TABLE xTmpUnconvertedTemplates
AS
SELECT TemplateID , CAST(FieldDefs AS TEXT) AS FieldDefsTxt
FROM SourceTemplateTable
;
DELETE FROM xTmpUnconvertedTemplates -- delete the already Hybrid templates or !MyFreeForm3 templates
WHERE
    (
     FieldDefsTxt LIKE '%<FieldName>ForceFF</FieldName>%' -- ForceFF is added by Hybrid
     OR
     FieldDefsTxt LIKE '%<FieldName>ShortPage</FieldName>%' -- ShortPage is added by !MyFreeForm3 which does not need conversion
     )
;
 
-- Convert Source Template FieldDefs to Hybrids
UPDATE SourceTemplateTable
SET FieldDefs =
CAST(
     REPLACE(CAST(FieldDefs AS TEXT), '<Fields>', '<Fields><Field><FieldName>Footnote</FieldName><DisplayName>Footnote (FF)</DisplayName><Type>Text</Type><Hint>footnote sentence from Free Form source, without the source detail</Hint><LongHint/><CitationField>False</CitationField></Field><Field><FieldName>ShortFootnote</FieldName><DisplayName>Short Footnote (FF)</DisplayName><Type>Text</Type><Hint>short footnote sentence from Free Form source sans source detail</Hint><LongHint/><CitationField>False</CitationField></Field><Field><FieldName>Page</FieldName><DisplayName>Page (FF)</DisplayName><Type>Text</Type><Hint>page value from Free Form source, originally the source detail</Hint><LongHint/><CitationField>True</CitationField></Field><Field><FieldName>Bibliography</FieldName><DisplayName>Bibliography (FF)</DisplayName><Type>Text</Type><Hint>bibliography sentence from FF source</Hint><LongHint/><CitationField>False</CitationField></Field><Field><FieldName>ForceFF</FieldName><DisplayName>FORCE FF</DisplayName><Type>Text</Type><Hint>Any value will force Free Form (FF) sentences</Hint><LongHint>If left empty, the sentences will be generated from the template fields below this field. If non-empty, the sentences will come from the Free Form-like fields above this one or designated (FF).</LongHint><CitationField>False</CitationField></Field>')
     AS BLOB)
WHERE TemplateID IN (SELECT TemplateID FROM xTmpUnconvertedTemplates)
;
 
/*
Some templates already had a [Page] field so now they have two and one must go
The only one we can be sure of having a constant pattern is the one we just added
so we delete it.
*/
UPDATE SourceTemplateTable
SET FieldDefs =
CAST(REPLACE(CAST(FieldDefs AS TEXT)
     ,'<Field><FieldName>Page</FieldName><DisplayName>Page (FF)</DisplayName><Type>Text</Type><Hint>page value from Free Form source, originally the source detail</Hint><LongHint/><CitationField>True</CitationField></Field>'
     ,''
     )
     AS BLOB
    )
WHERE TemplateID
IN(
   SELECT TemplateID
   FROM  xTmpUnconvertedTemplates
   WHERE FieldDefsTxt LIKE '%<DisplayName>Page</DisplayName>%' -- template already having a [Page] field
   )
;
 
/*
Convert Source Template sentence templates to Hybrid
e.g. Footnote: <? [ForceFF]|[Footnote], [Page].|--originalfootnote--|[Footnote], [Page].>>
*/
UPDATE SourceTemplateTable
SET
Footnote =
REPLACE(Footnote, Footnote, '<? [ForceFF]|[Footnote], [Page].|' || Footnote || '|[Footnote], [Page].>')
,
ShortFootnote =
REPLACE(ShortFootnote, ShortFootnote, '<? [ForceFF]|[ShortFootnote], [Page].|' || ShortFootnote || '|[ShortFootnote], [Page].>')
,
Bibliography =
REPLACE(Bibliography, Bibliography, '<? [ForceFF]|[Bibliography].|' || Bibliography || '|[Bibliography].>')
WHERE TemplateID IN (SELECT TemplateID FROM xTmpUnconvertedTemplates)
;
 
/*
Done.
*/

Set Hybrids to Free Form for Export to GEDCOM

By using well both the (FF) fields and the more specific template fields for each source, you have both a quality sentence conforming to Evidence guidelines for use in reports AND a sentence that exports well to GEDCOM from the same source. It’s necessary to Force the (FF) sentence to be outputted, otherwise, the default is the source-specific template sentence. One does so by putting any value into the FORCE FF field. This query looks for any sources that have a non-empty Footnote (FF) sentence and an empty FORCE FF field and puts the value “GEDCOM” into the latter. Any sources already forced to output the Free Form sentence are untouched. Sources that have empty Free Form Footnote fields are also left unchanged.

Download

Sources-HybridFF-SetFF-Export.sql

Script

-- Sources-HybridFF-SetFF-Export.sql
/*
2013-12-17 Tom Holden ve3meo
 
Sets Hybrid Free Form Templated Sources to output the Free Form footnote,
if not already so set, using the value "GEDCOM" in the ForceFF field.
 
The complementary script sets those Hybrid templates with "GEDCOM" in the
ForceFF field to output the templated sentence, as would be desirable for all
other destinations, such as reports.
*/
 
DROP TABLE IF EXISTS xTmpHybridSourcesSetFF
;
CREATE TEMP TABLE IF NOT EXISTS xTmpHybridSourcesSetFF
AS
SELECT SourceID, CAST(FIELDS AS TEXT) AS FieldsTxt
FROM SourceTable
WHERE CAST(FIELDS AS TEXT) LIKE '%<Field><Name>ForceFF</Name><Value/></Field>%' -- empty FORCE FF field
AND CAST(FIELDS AS TEXT) LIKE '%<Field><Name>Footnote</Name><Value>%'  -- non-empty Footnote (FF) field
;
 
UPDATE OR REPLACE xTmpHybridSourcesSetFF
SET FieldsTxt =
REPLACE (
         FieldsTxt
         , '<Field><Name>ForceFF</Name><Value/></Field>'
         , '<Field><Name>ForceFF</Name><Value>GEDCOM</Value></Field>'
         )
;
 
UPDATE OR REPLACE SourceTable
SET FIELDS =
(
SELECT CAST(FieldsTxt AS BLOB) FROM xTmpHybridSourcesSetFF
 WHERE SourceTable.SourceID = xTmpHybridSourcesSetFF.SourceID
)
WHERE SourceID IN (SELECT SourceID FROM xTmpHybridSourcesSetFF)
;

Set Hybrids to Source-Specific Sentences (Unset Free Form) for Reports et al

After having run the above query to set sources to export the Free Form sentences, you will want to reset them back to outputting the source-specific or source template sentences for most other uses. This script looks for all the FORCE FF fields having the value “GEDCOM” and empties them. It won’t touch those sources that have some other value in the FORCE FF field – for example, you may have sources for which you prefer the Free Form sentences to those using the source-specific fields.

Download

Sources-HybridFF-UnSetFF.sql

Script

-- Sources-HybridFF-UnSetFF.sql
/*
2013-12-17 Tom Holden ve3meo
 
Sets Hybrid Free Form Templated Sources to output the templated or non-Free Form
sentences, if set to output Free Form by the value "GEDCOM" in the ForceFF field,
as would be desirable for all other destinations, such as reports.
 
The complementary script sets those Hybrid templates with an empty ForceFF field
to output the Free Form sentence by adding setting the ForceFF field to "GEDCOM".
*/
 
DROP TABLE IF EXISTS xTmpHybridSourcesUnSetFF
;
CREATE TEMP TABLE IF NOT EXISTS xTmpHybridSourcesUnSetFF
AS
SELECT SourceID, CAST(FIELDS AS TEXT) AS FieldsTxt
FROM SourceTable
WHERE CAST(FIELDS AS TEXT) LIKE '%<Field><Name>ForceFF</Name><Value>GEDCOM</Value></Field>%'
;
 
UPDATE OR REPLACE xTmpHybridSourcesUnSetFF
SET FieldsTxt =
REPLACE (
         FieldsTxt
         , '<Field><Name>ForceFF</Name><Value>GEDCOM</Value></Field>'
         , '<Field><Name>ForceFF</Name><Value/></Field>'
         )
;
 
UPDATE OR REPLACE SourceTable
SET FIELDS =
(
SELECT CAST(FieldsTxt AS BLOB) FROM xTmpHybridSourcesUnSetFF
 WHERE SourceTable.SourceID = xTmpHybridSourcesUnSetFF.SourceID
)
WHERE SourceID IN (SELECT SourceID FROM xTmpHybridSourcesUnSetFF)
;