Group – Persons with Duplicate Events #namedgroup #events #date #duplicates

2023-02-19 Version for #RM8 added.

  • Creates a Named Group in the open database with the name “SQL: Duplicate Events” if one does not already exist
  • Removes all members from that named group
  • Populates that named group with persons having “duplicate” events

Duplication is declared if a pair of events:

  • (have dates for which the span of calendar years (ignoring month and day) is less than a parameter (2 in the script) OR
  • do not have dates but are fact types that use dates) AND (are both set or both not set as Primary events)

Duplication is resolved in RM by:

  • deleting an event (you may want to incorporate an alternative date in the date or notes of the remaining event) OR
  • setting one event only of the pair as Primary

To refresh the named group’s membership list to reflect corrections you may have made to resolve duplication of events, rerun the query to regenerate the RootsMagic group table and then, in RootsMagic, select a different group and then re-select the “SQL: Duplicate Events” group. You should see that names have disappeared from the group, provided you have corrected all the duplicate events for each person you worked on. The Quick Groups feature in RM7 and the Groups list in RM8 allow you to unmark a member from the group right from the Edit Person screen making it unnecessary to rerun the script unless you missed doing so.

To vary the parameter controlling how sensitive the query is to date differences, change this line at the end of the script:

WHERE ABS(SUBSTR(Date1,4,4) - SUBSTR(Date2,4,4)) <2 -- this controls how tight the year match must be.
-- in future, the date comparison could call on a date calculator that could compute the difference in days
 
)
; --END of SCRIPT

A setting of “<2” permits two consecutive years which allows Dec 31, 1900 and Jan 1, 1901 to be considered duplicates as it does for Jan 1, 1900 and Dec 31, 1901. A setting “<1” requires the events to be in the same year.

Group-PersonsWithDuplicateEvents.sql rev 2018-07-07 bug fixed when no other existing group

RM8 Version 2023-02-19

Nickname Manipulations #nicknames

Why?

RootsMagic has three different storage mechanisms for Nicknames:

  1. Nickname field in each Name record
  2. In quotation marks or parentheses in the Given field of each Name record
  3. As above in a Name record of type “Nickname”

Different outputs from RootsMagic have differing usage of these. Box charts cannot use Alternate Names and only include the primary name Nickname field if Prefix and Suffix are also included. Sentence templates for facts other than the Alternate Name type cannot access Alternate Names, cannot parse the nickname embedded in the primary Given field but can use the primary Nickname field. So there are outputs where it may be desirable that nicknames be embedded with the given name and others where it should not be. And as a database is built over time from a variety of sources, there may be inconsistency in the way nicknames are stored within the database.

This page offers a set of scripts that can manipulate where the nickname values are stored, globally and quickly, to optimise for a desired output and to make consistent.

Combine | Split Nickname and Given Names

NickGivenSplitCombine.png
These two scripts can swap the nickname between the NickName field and the Given Names field. The splitter works only with quotation marks delimiting the nickname in the Givens field; the combiner puts the nickname in quotes in the Givens field. You can use RootsMagic Search and Replace function on the Given names to convert all instances of parentheses to double quotation marks.
NickGivenSplit.sql
NickGivenCombine.sql

Ambivalent Nicknames

This script returns Record Number and Names of Persons who have both an embedded Nickname and a value in the Nickname field. These cannot be processed by NickGivenCombine.sql and NickGivenSplit.sql and should be resolved in the database with one or the other but not both.

Includes those having parentheses to denote nickname in the Given field instead of quotes; parentheses not supported by other scripts in this series.

NicknameAmbivalent.sql

Conflict between Primary and Alternate Nicknames

This script returns the Record Number and Nicknames of Persons who have both an Alternate Nickname and a Primary name with an embedded nickname or non-empty NickName field and the nickname values in the corresponding fields do not match. Use RIN to find the person in RootsMagic Explorer to revise as needed.
NickAltNameConflict.sql

UNDER CONSTRUCTION
-possibly converters between the nickname fields of Primary Name and Alternate Name of type Nickname but this would be complex.

Media Users List Query #media #rm8

Update

As of RootsMagic 8 Preview 7.9.180, the traceability of where an image is used has changed little from what was described below for RM4. The need for an outboard query such as this is as high in early 2021 as it was in 2010. With some changes in tables from RM7, the Media-UsersList4.sql query does not run on RM8. Pat Jones revised it and applied a technique she describes at Common Table Expressions – The Building Blocks of SQL to streamline it:

Media-UsersList4-RM8.sql

For a version that works with databases from RM5 to RM7:
Media-UsersList4-RM7.sql 

Introduction

RootsMagic 4 falls short in handling media in several ways. One serious one is its inability to report fully all the ‘owners’ of the media, i.e., its terminology to describe what person, couple, event, source, citation, place or place detail to which a media item is attached, in sufficient detail that the attachment can be found. Trace-ability is essential to be able to make changes to an attachment’s properties, i.e., its caption, description, reference number, date, sort date, primary and scrapbook inclusion flags. RM4’s Report > Lists > Multimedia List does not even report usage by Sources or Source Details and fails to report the Place to which a Place Detail, having media attached, belongs. Similarly, the Media Gallery > Tools > “Show where this media is used” results are incomplete for media attached to Source Details or to Place Details, giving little or no clue to how you would navigate to the citation or Place where you could then drill down to the Media and edit the caption or other properties.

Here’s an example of what I mean:

Owner according to Media Gallery > Tools > “Show where this media is used”

Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete) (citation)

How can one possibly find the citation without at least the RIN or name of the person?

Users according to Media-UsersList query

(citation) WICKENS, Martha Jane "Jenny"-208 : RESIDENCE (FAMILY) ca 1891 : HOLDEN, Matthew Edwin-149 citing Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete)
(citation) HOLDEN, Matthew Edwin-149 : RESIDENCE (FAMILY) ca 1891 : WICKENS, Martha Jane "Jenny"-208 citing Cens: 1891 Can: Ancestry.ca - FF2 (ff incomplete)

Both RM4 and the query show that the media file is attached to a citation but the new query provides explicit guidance to the citation via not only the name and RIN of the person but also the fact name and date so that the citing fact can be quickly found and its Citation Manager opened. The source name is shown (that’s all the Media Gallery shows apart from the caption and full filename) so it can be readily selected from the Citation Manager and the Source Detail Media Album opened therefrom and the media item selected by file name for the editing of its properties. That is the only way RM4 lets you get to the item’s properties; obviously, a road map is necessary. This query shows the route.

Download

Media-UsersList.sql – original query, easier to modify, fast on smaller databases, bogs down on large ones
Media-UsersList3.sql RMtrix_tiny_check.png – fast even on very large databases, larger sql file, harder to modify at the base SELECTs.
Media-UsersList4.sql – as ver 3 but revised for RM5 and 6; prior versions work with RM4.

Screenshots

With Thumbnail Images

Using SQLite Expert Personal, make sure the following setting is checked for the thumbnail image to be shown:

Tools > Options > Data:

  • Results-Display results in grid
  • Grids-Show images
Media-UsersList_Screenshot_from_Expert.PNG
SQLite Expert Personal can display the thumbnail images stored in the database. A subset of the available columns is displayed to reduce clutter. Unfortunately, column widths are not remembered on re-executing the same query. Also, sorting on column heading does not work unless the ORDER BY clause at the very end of the query is deleted.

Without Thumbnail Images

Media-UsersList_Screenshot_from_Spy.PNG
Screenshot from SQLiteSpy which has no image display capability but does remember column widths within the same session and supports sorting on column regardless of an ORDER BY in the query.

Discussions & comments from Wikispaces site


ve3meo

 

Comment: “Download”

ve3meo
03 September 2018 21:38:03

ve3meo Nov 14, 2011

This query can bog down on very large databases because it produces a very large set of intermediate results that must be stored in memory. It is being (slowly) revised to produce many small sets of intermediate results so that it will be able to handle large databases efficiently. This is not a trivial re-write.
ve3meo Nov 14, 2011

Re-write done: Media-UsersList3.sql. On a large database that was taking 37 to 120 seconds to display results, the time has dropped to 1-5s, depending on the SQLite manager, what else is going on, how much memory is available.The key was to structure it to minimise the size of intermediate results that are then JOINed with another result set.

Citations Invisible – Reveal and Revert #citations #sources #ftm2014 #ftm2012 #ancestrycom #gedcom

With the flurry of users of Family Tree Maker looking to migrate their database to RootsMagic, an outcry has gained volume because of the disappearance of some citations. This problem was described here in 2012 in Citations Invisible Revealed which offered two scripts, both of which are available in the RMtrix utility, one to find such citations for the Primary (Preferred Name), the other to convert them to citations for the Person. That was an irreversible conversion. In the hope that RootsMagic will eventually support citations for the Primary (Preferred) Name, this pair of scripts provides both conversion and reversion. As of 2015-12-24, RootsMagic is expected to issue an update that will do a similar conversion and an upgrade that will support citations for the Primary (Preferred) Name and revert these conversions. However, it will use a different value in the flag to make the citations reversible than was used in the initial scripts, necessitating these revised versions to be compatible.

Citations, Invisible – Convert to Personals + Flagged.sql rev 2015-12-24 Flags field now set bitwise on 1st bit instead of to 7
Citations, Invisible Revealed – Revert – Bitwise.sql 2015-12-24 Unsets 1st bit of Flags field instead of setting to 0

Caveat! To make it reversible, the converter uses the apparently unused Flags column in the CitationTable. The initial converter dated 2015-12-19 set this value to 7, corresponding to the original OwnerType value for the Citation Table record; the value that RootsMagic will use for this purpose will be the 1st bit set to 1, allowing the Flags field to be used for many bitwise flags. I cannot predict what the consequences might be for those records having a flag value of 7 set by the initial converter. If you used the now-deprecated, initial version of Citations, Invisible – Convert to Personals + Flagged.sql, it would be advisable to run this complementary script to revert them to the primary Name fact: Citations, Invisible Revealed – Revert.sql Then follow up with the revised converter above or wait for the RootsMagic update to do the same thing.

James Clouse posted this video to the RootsMagic Users Facebook Group showing how to use SQLiteSpy to open your database and run the conversion script. Click on the settings and select HD to avoid fuzziness.

Citations Invisible Revealed #citations #ancestrycom

N.B. This is now the background page for a pair of revised scripts that supersede the conversion script here and make the process reversible. See Citations Invisible – Reveal and Revert.

First reported by Paul Harris in the RootsMagic Forums in the discussion Unused Citations after Import, we have concluded that there is a bug in RootsMagic 5.0.4.1 and, possibly, earlier versions that suppresses or fails to include a certain type of citation from appearing in key user screens and reports. Yet these citations are generated by RootsMagic, can be seen in the Source List report and survive a drag and drop transfer to a new database.

Citations, Invisible – List.sql RMtrix_tiny_check.png This query will show whether there are any such ‘invisible’ citations in a RootsMagic database.
Citations, Invisible – Convert to Personals.sql RMtrix_tiny_check.png This query will convert these ‘invisible’ citations to a type that is visible in the RootsMagic program.

The ‘invisible’ citations covered by these queries are for the Primary Name of a person and are converted to a Personal citation to become visible. We think an Ancestry citation that “provides evidence for Name” is exported as a citation for the “Preferred” Name, when there are Alternate Names, not for the Person. If there are no Alternate Names, we surmise that Ancestry exports the citation against the Person (top level in GEDCOM, Name is lower); these are visible. Citations for Alternate Names are visible and correctly placed in the Edit Person screen. Since there is no other place for a Primary Name citation to go than beside the name at the top of the Edit Person list and since that is where Personal citations do get flagged, then it is reasonable to convert the ‘invisible’ citations to Personal ones.

On import, RootsMagic assigns an OwnerType = 7 in the CitationTable for citations for Names, be they Preferred or Alternate, signifying that the corresponding CitationTable.OwnerID points to a unique NameID in the NameTable. Each unique NameID has a set of names for a Person in the PersonTable whose unique PersonID corresponds to the NameTable.OwnerID. For a given NameTable.OwnerID (or Person), there should be but one record with the IsPrimary column set to 1 – this is the Primary Name; all other records in the set have IsPrimary = 0, signifying they are Alternate Names. The List query lists all citations of Type 7 that point to a Primary Name.

For Personal citations, OwnerType = 0 in the CitationTable, signalling to RootsMagic to match the CitationTable.OwnerID with PersonTable.PersonID and that this is a Personal citation.

The Convert to Personals query replaces the CitationTable.OwnerID with the PersonID owning the NameID pointed to by the citation and sets OwnerType to 0 for only those citations for a Primary Name (OwnerType = 7, linked NameTable.IsPrimary = 1 or True).

Dates – Same Day Sort Order #date #sortdate #update

#RM10 compatible

Problems

Some RootsMagic users bemoan the fact that events having the same date are displayed and reported in an unnatural or illogical order, having more to do with the order they were entered. Burial before Death or Christen before Birth are often mentioned. RootsMagic does provide the user with the option to modify the SortDate date to establish the desired order.

RM’s proprietary SortDate is lost in an export to another program; any import into RM from anything other than a RM database or an RM generated GEDCOM with the Extra Details (RM specific) box checked will be assigned SortDates matching the event date resulting in the probable loss of the desired order. So manual control over the SortDate is of no comfort to those who move their database between different programs to take advantage of their respective strengths. An enhancement to provide better results for the order of same day events has long been on the RootsMagic Wish List.

Workaround

Meanwhile, here are two SQLite queries that can re-order all same day events of limited types in seconds with results as shown in the screenshots below. They both exploit the SortDate feature that values of the form date-n, where n is an integer from 1 to 4999, are sorted in ascending order.

Dates-SameDaySortOrder.png
Before and after the query was applied.

SortDateSameDayOrder.sql

SortDateSameDayOrder.sql, currently addresses the following in the order listed: Birth, Christen, Baptism, Death, Cremation and Burial. Others could be added. Should work with any SQLite manager on both RM4 and RM5 databases. Currently deprecated because it changes other facts on the same day; left here because its comparative simplicity may help to understand its successor below.

/* SortDateSameDayOrder.sql
   2011-12-19 ve3meo
   Alters SortDates of Birth, Christen, Baptism, Death, Cremation and Burial
   to a natural order when any pair or more occur on the same date.
   Could be extended to order other facts also. SortDates are effectively assigned
   (by arithmetical offsets) an absolute suffix -1, -2, ... related to the FactType.
   Affects only those events whose SortDates correspond to the Fact Date, as computed
   by a Date encoding algorithm. The algorithm does not handle Date modifiers so not all
   Event dates are handled, e.g. "Bef 1960".
   2011-12-20 order of Cremation and Burial corrected
*/
 
UPDATE EventTable
SET SortDate = SortDate-6692012023*(EventType
  IN (1,3,7,2,4,5))  -- list of FactTypes we want to sort, in no particular order except this corresponds to the desired order
  +1048576*(SUBSTR('1426503',EventType,1)-1) -- the substr maps the FactType to its order
WHERE EventID
IN (SELECT EventID FROM EventTable
    INNER JOIN
    (SELECT -- matching dates
     SortDate, OwnerID, COUNT()-1 AS Matches FROM EventTable
     WHERE OwnerType = 0 AND EventType IN (1,3,7,2,5,4)
     AND
     SortDate =   -- equals encoded event Date (if not a match, suggests that user has modified SortDate so don't touch it)
       (CASE
        WHEN DATE LIKE '.%'
        THEN 1
        ELSE Substr(DATE,3,5) END
        +10000
        )*562949953421312
        + Substr(DATE,8,2)*35184372088832
        + Substr(DATE,10,2)*549755813888
        + 17178820620
      GROUP BY SortDate, OwnerID, OwnerType
     )
     USING (OwnerID, SortDate)
     WHERE Matches
    )
;

SortDateSameDayOrderCustom.sql

SortDateSameDayOrderCustom.sql RMtrix_tiny_check.png
Goes to the next level, supporting the ranking of any number of facts, be they standard or custom. It also corrects an oversight in SortDateSameDayOrder.sql and does not touch unranked facts on the same date. Requires the use of a fake RMNOCASE collation or possible revision with COLLATE NOCASE phrases where FactTypeTable.Name is used. Because SortDate supports up to date-4999, this query could be modified to have a multiplier of the rank, thus leaving room for other events on the same date to be manually added with a SortDate lying between those assigned by the query, without having to revise the existing SortDates.

/* SortDateSameDayOrderCustom.sql
   2011-12-20 ve3meo
   Alters SortDates of any set of Fact types
   to a natural order when any pair or more occur on the same date.
   Could be extended to order other facts also. SortDates are effectively assigned
   (by arithmetical offsets) an absolute suffix -1, -2, ... related to the rank of the FactType.
   Affects only those events whose SortDates correspond to the Fact Date, as computed
   by a Date encoding algorithm. The algorithm does not handle Date modifiers so not all
   Event dates are handled, e.g. "Bef 1960".
*/
DROP TABLE IF EXISTS TmpFactOrder
;
CREATE TEMP TABLE IF NOT EXISTS TmpFactOrder
(Rank INTEGER PRIMARY KEY, FactName TEXT)
;
 
/* list of Fact Names, standard and custom, to be sorted, in rank order.
   Revise the list to suit your needs */
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Birth');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Christen');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Baptism');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Death');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Funeral');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Cremation');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Burial');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Obituary');
INSERT INTO TmpFactOrder (Rank, FactName) VALUES (NULL, 'Memorial');
/* revise SortDates */
UPDATE EventTable
SET SortDate = SortDate
  -6692012023  -- this offset goes to Date-1 if the event is a ranked event
  *( (
      SELECT Rank
      FROM TmpFactOrder,
           FactTypeTable
      WHERE FactName LIKE Name
      AND FactTypeID = EventType
      )>0
    )
  +1048576  -- this offset adds steps of 1 to Date-1 multiplied by (rank-1)
  *( (
      SELECT Rank
      FROM TmpFactOrder,
           FactTypeTable
      WHERE FactName LIKE Name
      AND FactTypeID = EventType
      )-1
    ) -- maps the FactType to its order
WHERE EventID
IN (SELECT EventID FROM EventTable
    INNER JOIN
    (SELECT -- matching dates
     SortDate, OwnerID, COUNT()-1 AS Matches
     FROM EventTable
     INNER JOIN FactTypeTable
     ON EventType = FactTypeID
     WHERE EventTable.OwnerType = 0
     AND Name IN (SELECT FactName FROM TmpFactOrder)
     AND
     SortDate =   -- equals encoded event Date (if not a match, suggests that user has modified SortDate so don't touch it)
       (CASE
        WHEN DATE LIKE '.%'
        THEN 1
        ELSE Substr(DATE,3,5) END
        +10000
        )*562949953421312
        + Substr(DATE,8,2)*35184372088832
        + Substr(DATE,10,2)*549755813888
        + 17178820620
      GROUP BY SortDate, OwnerID, EventTable.OwnerType
     )
     USING (OwnerID, SortDate)
     INNER JOIN FactTypeTable
     ON EventType = FactTypeID
     WHERE Matches
     AND EventTable.OwnerType = 0
     AND Name IN (SELECT FactName FROM TmpFactOrder)
    )
;

Discussions & comments from Wikispaces site


chuckfilteau

Alternate method using the TmpFactOrder table

chuckfilteau
05 November 2016 18:50:42

Thanks for the algorithm and SQL code to set the SortDate.
From it, I created the TmpFactOrder table, but I added two additional columns (both integers)
FT – for the FactType that matched the FactName
ADJ – to pre-calculate each adjustment for each Rank
I then coded two simple update statement to update this table

◦Rationale – do the work only once, in this table, rather than redoing the work in SQL for each fact that had duplicate names **
UPDATE TmpFactOrder SET FT = (Select FactTypeID from FactTypeTable where FactName = Name);
UPDATE TmpFactOrder SET ADJ = ((Rank-1)*1048576) – 6692012023;

Finally, I came up with a much simpler, easier to understand, SQL statement that updated the SortDate for all events that had duplicate dates (for standard single date values)
Here is that statement:
UPDATE EventTable
SET SortDate=
((Substr(DATE,3,5) +10000)*562949953421312 — year
+ Substr(DATE,8,2)*35184372088832 — month
+ Substr(DATE,10,2)*549755813888 — day
+ 17178820620 — constant (time?)
+ (Select ADJ from TmpFactOrder — Rank adjustment (-1 to -20)
where EventType = FT)
)
where EventID in
(Select distinct E1.EventID — get the list of IDs that need to be changed
from EventTable as E1
JOIN EventTable as E2
Using (OwnerID,Date)
where E1.OwnerType=0
AND E1.[Date] like “D.%” — only events that have a single standard date
AND E1.EventType IN (Select FT from TmpFactOrder)
AND E1.EventID <> E2.EventID — DISCARD MATCHES ON THE SAME ROW
)
;
/* ALL DONE */


thejerrybryan

Minor “problem” with the handling of sort dates

thejerrybryan
11 September 2017 15:35:28

I am playing around with Tom’s original SortDate code rather than with chuckfilteau’s SortDate code. The code is very tricky (to me, at least), and the “problem” I have identified is most likely in myself rather than in Tom’s code.

I am using SQLite to add a Parents fact to each person who has parents. Having done so, I want the sort dates set such that the sort date for the Birth fact becomes birthdate_fact_date-1 and the sort date for the Parents fact becomes parents_fact_date-2, where birthdate_fact_date and parents_fact_date are the same date. So in addition to requiring a person actually to have parents before adding a Parents fact, my SQLite code also checks for the presence of a Birth fact and only adds the Parents fact if there is a Birth fact. For the newly added Parents fact, my code sets the Date, SortDate, and EditDate to be the same as for the Birth fact with other Parents fact fields being set appropriately for the Parents fact.

Having run my code to add the Parents fact, I copy Tom’s SortDate code, add my Parents fact to the list of facts, and run my copy of Tom’s code. All is mostly well, but all is not completely well. For an example when all is well, if the original birth date and sortdate is 18 May 1819, then my code sets the Parents fact date and sortdate to 18 May 1819. Tom’s code sets the Birth fact sortdate to 18 May 1819-1 and the Parents fact sortdate to 18 May 1819-2. Perfect.

For an example of when all is not well, I have a person with a birth date of 1790 and a sort date of 1 Jul 1790. So my newly added Parents fact also gets a date of 1790 and a sort date of 1 July 1790. Under these conditions, Tom’s code does not adjust the sort dates to 1 Jul 1790-1 and 1 Jul 1790-2, respectively. It thinks I have already set sort dates manually that should not be overridden. If before running Tom’s code, I change both sort dates to 1790, then Tom’s code changes the sort dates to 1790-1 and 1790-2, respectively. Perfect, except that all the sort dates in my database seem to be 1 Jul 1790 for year dates and something like 15 Mar 1790 for month dates like Mar 1790. And curiously, when I add a new year only date by hand, RM sets the sort date just to the year and not to 1 Jul of the year.

For another example of when all is not well, I have person with a birth date of Abt 1725 and a sort date of 1 Jul 1725. My code adds a Parents fact with a date of Abt 1725 and a sort date of 1 Jul 1725. Again, Tom’s code does not adjust the respective sort dates. I may not be understanding the code correctly, but I think the problem is not with the “Abt” modifier but again it’s because the code sees both events as having a 1 Jul 1725 sort date. If I manually change both sort dates to just plain 1725, then Tom’s code is not bothered by the “Abt” modifier and it does change the sort dates to 1725-1 and 1725-2, respectively. Again, this is perfect except that all my Abt dates for years have sort dates of 1 Jul for the appropriate year. I remember the use of 1 Jul sort dates being a standard RM technique for year only dates and my database is full of them, but now if I add a new year only fact by hand, the sort date is the year only without the 1 Jul. Has something changed in RM and I didn’t notice?

I thought I had come to a sufficient understanding of the sort date code to adjust it to meet my needs, but so far my efforts have failed. I can’t get it to override a Birth sort date of 1 Jul 1790 and a Parents sort date of 1 Jul 1790. I want the dates to become 1 Jul 1790-1 and 1 Jul 1790-2, respectively. Therefore, any advice would be most gratefully accepted.


thejerrybryan

thejerrybryan
11 September 2017 16:37:37

Aha (I think)!

I commented out the following code, and it seems to have solved my problem. I was trying to make changes within the commented out code, and I just needed to get rid of the commented out code entirely.

/* AND
SortDate = — equals encoded event Date (if not a match, suggests that user has modified SortDate so don’t touch it)
(CASE
WHEN DATE LIKE ‘.%’
THEN 1
ELSE Substr(DATE,3,5) END
+10000
)*562949953421312
+ Substr(DATE,8,2)*35184372088832
+ Substr(DATE,10,2)*549755813888
+ 17178820620 */

I would be interested in opinions concerning the following observation. It seems to me that commenting out the code in question might be a good idea for the general use case, not just for my specific use case. Which is to say that even with this code removed from the script, the script still does not modify sort dates that are different. For example, if the RM user has established a birth sort date of 1 Jul 1850-1 and a death sort date of 1 Jul 1850-2, then the script will not modify the sort dates even with the code commented out as indicated. The sort dates are not identical and are therefore are prevented from matching by INNER JOIN’s. They don’t need to be prevented from matching by the code that has been commented out. At least that’s the way it seems to me, and that’s what I’m going to go with for now.

Recursive Queries – Ancestors and Descendants #recursive #ancestors #descendants

As of Feb 3, 2014 and version 3.8.3, SQLite introduced “Recursive common table expressions (that) provide the ability to do hierarchical or recursive queries of trees and graphs, a capability that is not otherwise available in the SQL language”. (http://www.sqlite.org/lang_with.html) This means it is now possible to generate ancestry or descendancy lists within SQLite, without having to resort to a higher level language. These lists can be unlimited in length, unlike queries based on earlier versions of SQLite, which required a SELECT statement for each generation (e.g. the 2011 Ancestors Query). The WITH RECURSIVE syntax is much shorter than cascading like queries and executes much faster (60ms vs 300ms, for example). While the syntax is challenging to understand, an error or change needs to be dealt with in one or two spots, not in every SELECT in a cascade. Here’s a helpful guide to gain understanding: The Simplest SQLite Common Table Expression Tutorial.

A first attempt at a recursive query was reported on Ancestors Query – Recursive. I then proceeded to develop the complement – a recursive Descendants query. Because the two are complementary and will benefit from parallel development, and because recursion is such a significant step forward in the power of SQLite, a page focussed on recursion as it applies to the two essential genealogical queries seems appropriate. This page presents an enhanced Ancestors query and its complementary Descendants query and will, hopefully, see further development.

rev 2015-11-24 Descendants: filter added for sex so that a birth only relationship for female descendants of a woman would be those sharing mtDNA from her; the complementary male line is also provided which would those sharing the Y-STR chromosome. Note that RootsMagic has in its Select People dialogs tools to mark/unmark people by genetic lines.

Ancestors

This query is fundamentally the same as Ancestors Query – Recursive with the addition of named parameters which the user defines at run-time:

  1. $BirthOnly(YN): enter “Y” or “y” to restrict the lineage to birth relationships (bloodlines); entry of anything else will include all relationships (adoptive, step, etc.)
  2. $Person(RIN): enter the record number of the person whose list of ancestors is wanted

The result is a list of the record numbers of the ancestors under the heading AncestorID and does not include the starting RIN.
AncestorsRecursive.sql

-- AncestorsRecursive.sql
/*
2014-04-14 Tom Holden ve3meo
 
Generates the list of RINs for the ancestors of a person.
 
Requires support not only for SQLite 3.8.3 or later but also
for named parameters for user input of the RIN of the starting person
and choice of birth only or all relationships.
 
Developed and tested with current SQLite Expert Personal 3.5.36.2456
 
Uses the WITH RECURSIVE syntax introduced in SQLite 3.8.3 2014-02-03
modelled on http://www.sqlite.org/lang_with.html example
and complement of DescendantsRecursive.sql
*/
 
WITH RECURSIVE
  parent_of(ChildID, ParentID) AS
    (SELECT PersonID, FatherID AS ParentID FROM PersonTable
       LEFT JOIN ChildTable ON PersonID=ChildTable.ChildID
       LEFT JOIN FamilyTable USING(FamilyID)
       WHERE
         CASE $BirthOnly(YN)
         WHEN 'Y' OR 'y' THEN RelFather=0
         ELSE 1
         END
         --RelFather=0 --birth father (ELSE WHERE 1 to include all relationships)
     UNION
     SELECT PersonID, MotherID AS ParentID FROM PersonTable
       LEFT JOIN ChildTable ON PersonID=ChildTable.ChildID
       LEFT JOIN FamilyTable USING(FamilyID)
       WHERE
         CASE $BirthOnly(YN)
         WHEN 'Y' OR 'y' THEN RelMother=0
         ELSE 1
         END
         --RelMother=0 --birth mother (ELSE WHERE 1 to include all relationships)
     ),
  ancestor_of_person(AncestorID) AS
    (SELECT ParentID FROM parent_of
       WHERE ChildID=$Person(RIN) --enter RIN of starting person at runtime
     UNION --ALL
     SELECT ParentID FROM parent_of
       INNER JOIN ancestor_of_person ON ChildID = AncestorID)
SELECT AncestorID FROM ancestor_of_person, PersonTable
 WHERE ancestor_of_person.AncestorID=PersonTable.PersonID
;

Descendants

This query is a complement to the Ancestors query, structurally very similar, and with the same named parameters which the user defines at run-time:

  1. $BirthOnly(YN): enter “Y” or “y” to restrict the lineage to birth relationships (bloodlines); entry of anything else will include all relationships (adoptive, step, etc.)
  2. $Sex(MF?): enter one of “MmFf) to restrict the lineage to males or females only; anything else includes both.
  3. $Person(RIN): enter the record number of the person whose list of descendants is wanted

The result is a list of the record numbers of the descendants under the heading DescendantID and does not include the starting RIN.
DescendantsRecursive.sql rev 2015-11-24 filter added for sex so that a birth only relationship for female descendants of a woman would be those sharing mtDNA from her. The code following is the original query.

-- DescendantsRecursive.sql
/*
2014-04-14 Tom Holden ve3meo
 
Generates the list of RINs for the descendants of a person.
 
Requires support not only for SQLite 3.8.3 or later but also
for named parameters for user input of the RIN of the starting person
and choice of birth only or all relationships.
 
Developed and tested with current SQLite Expert Personal 3.5.36.2456
 
Uses the WITH RECURSIVE syntax introduced in SQLite 3.8.3 2014-02-03
modelled on http://www.sqlite.org/lang_with.html example
and complement of AncestorsRecursive.sql
*/
 
WITH RECURSIVE
  child_of(ParentID, ChildID) AS
    (SELECT PersonID, ChildTable.ChildID FROM PersonTable
       LEFT JOIN FamilyTable ON PersonID=FatherID
       LEFT JOIN ChildTable USING(FamilyID)
       WHERE
         CASE $BirthOnly(YN)
         WHEN 'Y' OR 'y' THEN RelFather=0
         ELSE 1
         END
         --RelFather=0 --birth father (ELSE WHERE 1 to include all relationships)
     UNION
     SELECT PersonID, ChildTable.ChildID FROM PersonTable
       LEFT JOIN FamilyTable ON PersonID=MotherID
       LEFT JOIN ChildTable USING(FamilyID)
       WHERE
         CASE $BirthOnly(YN)
         WHEN 'Y' OR 'y' THEN RelMother=0
         ELSE 1
         END
         --RelMother=0 --birth mother (ELSE WHERE 1 to include all relationships)
     ),
  descendant_of_person(DescendantID) AS
    (SELECT ChildID FROM child_of
       WHERE ParentID=$Person(RIN) --enter RIN of starting person at runtime
     UNION --ALL
     SELECT ChildID FROM child_of
       INNER JOIN descendant_of_person ON ParentID = DescendantID)
SELECT DescendantID FROM descendant_of_person, PersonTable
 WHERE descendant_of_person.DescendantID=PersonTable.PersonID
;

To Do’s

  1. Update previous queries that used the old cascaded Ancestors query, e.g., Ancestors Named Group
  2. Create new queries based on these simple lists of ancestors and descendants, e.g. Named Groups and Color Coding
  3. Investigate how one might add collateral lines
  4. How might one apply the ancestry query to clearing the Living flag for ancestors of persons who should be dead.
  5. Add generation number to results of ancestors and descendants – useful for reports
  6. Add FamilyID to results

Source Templates – Versatile Free Form Hybrids #sourcetemplates #sources

Introduction

This query introduces a radical variation on RootsMagic’s system of Source Templates. It combines Source Templates and Free Form properties into Versatile Free Form Hybrid Source Templates incorporating the best of both worlds. Sources built on older source templates will now have free form fields that can be used to create alternative sentences that will survive GEDCOM export better than the templated sentences. Sources built on Free Form are not changed but can now be converted losslessly to templated sources, the data values being transferred from the Free Form source to the Free Form-like fields in the Hybrid source. And new sources built on the Hybrid templates can begin either in Free Form or template style while the other can be created and edited from within the common Edit Source window.

This idea sprang from discussion on Jerry Bryan’s page Question with respect to Changing a Master Source to a Different Source Template.

Conversion from ordinary templates to Versatile Free Form Hybrid Source Templates

Screenshots

SourceTemplatesHybridFreeFormConvert-EditSrc1.PNG
Source using a Hybrid built-in source template showing the default output sentences coming from the original template fields, not the added (FF) fields.
SourceTemplatesHybridFreeFormConvert-EditSrc2.PNG
Same source but the FORCE FF field non-empty value causes the sentence outputs to come from the (FF) fields.

Restoring after Drag ‘n’ Drop to new database

Transfer of sources, based on built-in templates that have been so Hybridized, to another RM database initially appears to lose the Free Form fields. However, the data values were transferred and merely running the query on the other database will restore the transferred sources to the Hybrid state.

Conversion of Free Form sources to Hybrids

Conversion of Free Form sources to Hybrid templates is possible and not difficult. You need to replace the TemplateID value of 0 for specific sources in the SourceTable to the TemplateID value of the desired Hybrid template. The (FF) fields will be populated by the values of the Free Form source. If that template does not work out, you can readily change the source to another Hybrid template and not lose the (FF) values from which you parse values for the template fields.

Feedback

This query is EXPERIMENTAL – try it on a copy of your database. See if it works for you. Let me know what you think, what works, what doesn’t. For those who have used !MyFreeForm3, I suspect there will be complications that will be precluded in a later version – after all, I use it, but this initial version was developed on just the built-in source templates. who

Future Considerations

A later version will add the fields from !MyFreeForm3 for even better Free Form style sentences and another utility could toggle the ForceFF field to control which sentence template is used for export, i.e., Free Form for export, templated for reports and other outputs.

Download SQL Script

SourceTemplatesHybridFreeFormConvert.sql rev 2013-12-17: !MyFreeForm3 templates no longer converted; speed tweaks.

Script

--SourceTemplatesHybridFreeFormConvert.sql
/*
2013-12-16 Tom Holden ve3meo
2013-12-17 rev1. no longer converts !MyFreeForm3 templates
           rev2. possible speed up by reducing use of CAST
Marries source templates up with Free Form for the best of both worlds:
a) use the templated part as an input form to help draft a Free Form source
b) paste a citation from Ancestry, FamilySearch et al into the Free Form part to
refer to while populating the template fields.
c) toggle between sentences outputted from either the Free Form fields or the
templated fields
 
EXPERIMENTAL
The revised templates have added fields in Free Form style (FF) and a Force FF field
that determines whether the sentences outputted are from the Free Form fields or
the template fields.
 
N.B.: the built-in source templates, while converted, will not transfer
to another database. Because the target database has only the templated fields
in the template definitions, the Free Form fields will appear to be lost. Running
this utility on the target database will make them reappear with the values in
each source intact.
 
*/
 
-- backup SourceTemplateTable builtin templates (you could revert to this version
-- after executing the query the first time)
 
DROP TABLE IF EXISTS xSourceTemplateTableBak
;
CREATE TABLE IF NOT EXISTS xSourceTemplateTableBak AS
SELECT * FROM SourceTemplateTable
;
 
/*
-- Restore from backup
INSERT OR REPLACE INTO SourceTemplateTable
SELECT * FROM xSourceTemplateTableBAK
*/
 
DROP TABLE IF EXISTS xTmpUnconvertedTemplates
;
CREATE TEMP TABLE xTmpUnconvertedTemplates
AS
SELECT TemplateID , CAST(FieldDefs AS TEXT) AS FieldDefsTxt
FROM SourceTemplateTable
;
DELETE FROM xTmpUnconvertedTemplates -- delete the already Hybrid templates or !MyFreeForm3 templates
WHERE
    (
     FieldDefsTxt LIKE '%<FieldName>ForceFF</FieldName>%' -- ForceFF is added by Hybrid
     OR
     FieldDefsTxt LIKE '%<FieldName>ShortPage</FieldName>%' -- ShortPage is added by !MyFreeForm3 which does not need conversion
     )
;
 
-- Convert Source Template FieldDefs to Hybrids
UPDATE SourceTemplateTable
SET FieldDefs =
CAST(
     REPLACE(CAST(FieldDefs AS TEXT), '<Fields>', '<Fields><Field><FieldName>Footnote</FieldName><DisplayName>Footnote (FF)</DisplayName><Type>Text</Type><Hint>footnote sentence from Free Form source, without the source detail</Hint><LongHint/><CitationField>False</CitationField></Field><Field><FieldName>ShortFootnote</FieldName><DisplayName>Short Footnote (FF)</DisplayName><Type>Text</Type><Hint>short footnote sentence from Free Form source sans source detail</Hint><LongHint/><CitationField>False</CitationField></Field><Field><FieldName>Page</FieldName><DisplayName>Page (FF)</DisplayName><Type>Text</Type><Hint>page value from Free Form source, originally the source detail</Hint><LongHint/><CitationField>True</CitationField></Field><Field><FieldName>Bibliography</FieldName><DisplayName>Bibliography (FF)</DisplayName><Type>Text</Type><Hint>bibliography sentence from FF source</Hint><LongHint/><CitationField>False</CitationField></Field><Field><FieldName>ForceFF</FieldName><DisplayName>FORCE FF</DisplayName><Type>Text</Type><Hint>Any value will force Free Form (FF) sentences</Hint><LongHint>If left empty, the sentences will be generated from the template fields below this field. If non-empty, the sentences will come from the Free Form-like fields above this one or designated (FF).</LongHint><CitationField>False</CitationField></Field>')
     AS BLOB)
WHERE TemplateID IN (SELECT TemplateID FROM xTmpUnconvertedTemplates)
;
 
/*
Some templates already had a [Page] field so now they have two and one must go
The only one we can be sure of having a constant pattern is the one we just added
so we delete it.
*/
UPDATE SourceTemplateTable
SET FieldDefs =
CAST(REPLACE(CAST(FieldDefs AS TEXT)
     ,'<Field><FieldName>Page</FieldName><DisplayName>Page (FF)</DisplayName><Type>Text</Type><Hint>page value from Free Form source, originally the source detail</Hint><LongHint/><CitationField>True</CitationField></Field>'
     ,''
     )
     AS BLOB
    )
WHERE TemplateID
IN(
   SELECT TemplateID
   FROM  xTmpUnconvertedTemplates
   WHERE FieldDefsTxt LIKE '%<DisplayName>Page</DisplayName>%' -- template already having a [Page] field
   )
;
 
/*
Convert Source Template sentence templates to Hybrid
e.g. Footnote: <? [ForceFF]|[Footnote], [Page].|--originalfootnote--|[Footnote], [Page].>>
*/
UPDATE SourceTemplateTable
SET
Footnote =
REPLACE(Footnote, Footnote, '<? [ForceFF]|[Footnote], [Page].|' || Footnote || '|[Footnote], [Page].>')
,
ShortFootnote =
REPLACE(ShortFootnote, ShortFootnote, '<? [ForceFF]|[ShortFootnote], [Page].|' || ShortFootnote || '|[ShortFootnote], [Page].>')
,
Bibliography =
REPLACE(Bibliography, Bibliography, '<? [ForceFF]|[Bibliography].|' || Bibliography || '|[Bibliography].>')
WHERE TemplateID IN (SELECT TemplateID FROM xTmpUnconvertedTemplates)
;
 
/*
Done.
*/

Set Hybrids to Free Form for Export to GEDCOM

By using well both the (FF) fields and the more specific template fields for each source, you have both a quality sentence conforming to Evidence guidelines for use in reports AND a sentence that exports well to GEDCOM from the same source. It’s necessary to Force the (FF) sentence to be outputted, otherwise, the default is the source-specific template sentence. One does so by putting any value into the FORCE FF field. This query looks for any sources that have a non-empty Footnote (FF) sentence and an empty FORCE FF field and puts the value “GEDCOM” into the latter. Any sources already forced to output the Free Form sentence are untouched. Sources that have empty Free Form Footnote fields are also left unchanged.

Download

Sources-HybridFF-SetFF-Export.sql

Script

-- Sources-HybridFF-SetFF-Export.sql
/*
2013-12-17 Tom Holden ve3meo
 
Sets Hybrid Free Form Templated Sources to output the Free Form footnote,
if not already so set, using the value "GEDCOM" in the ForceFF field.
 
The complementary script sets those Hybrid templates with "GEDCOM" in the
ForceFF field to output the templated sentence, as would be desirable for all
other destinations, such as reports.
*/
 
DROP TABLE IF EXISTS xTmpHybridSourcesSetFF
;
CREATE TEMP TABLE IF NOT EXISTS xTmpHybridSourcesSetFF
AS
SELECT SourceID, CAST(FIELDS AS TEXT) AS FieldsTxt
FROM SourceTable
WHERE CAST(FIELDS AS TEXT) LIKE '%<Field><Name>ForceFF</Name><Value/></Field>%' -- empty FORCE FF field
AND CAST(FIELDS AS TEXT) LIKE '%<Field><Name>Footnote</Name><Value>%'  -- non-empty Footnote (FF) field
;
 
UPDATE OR REPLACE xTmpHybridSourcesSetFF
SET FieldsTxt =
REPLACE (
         FieldsTxt
         , '<Field><Name>ForceFF</Name><Value/></Field>'
         , '<Field><Name>ForceFF</Name><Value>GEDCOM</Value></Field>'
         )
;
 
UPDATE OR REPLACE SourceTable
SET FIELDS =
(
SELECT CAST(FieldsTxt AS BLOB) FROM xTmpHybridSourcesSetFF
 WHERE SourceTable.SourceID = xTmpHybridSourcesSetFF.SourceID
)
WHERE SourceID IN (SELECT SourceID FROM xTmpHybridSourcesSetFF)
;

Set Hybrids to Source-Specific Sentences (Unset Free Form) for Reports et al

After having run the above query to set sources to export the Free Form sentences, you will want to reset them back to outputting the source-specific or source template sentences for most other uses. This script looks for all the FORCE FF fields having the value “GEDCOM” and empties them. It won’t touch those sources that have some other value in the FORCE FF field – for example, you may have sources for which you prefer the Free Form sentences to those using the source-specific fields.

Download

Sources-HybridFF-UnSetFF.sql

Script

-- Sources-HybridFF-UnSetFF.sql
/*
2013-12-17 Tom Holden ve3meo
 
Sets Hybrid Free Form Templated Sources to output the templated or non-Free Form
sentences, if set to output Free Form by the value "GEDCOM" in the ForceFF field,
as would be desirable for all other destinations, such as reports.
 
The complementary script sets those Hybrid templates with an empty ForceFF field
to output the Free Form sentence by adding setting the ForceFF field to "GEDCOM".
*/
 
DROP TABLE IF EXISTS xTmpHybridSourcesUnSetFF
;
CREATE TEMP TABLE IF NOT EXISTS xTmpHybridSourcesUnSetFF
AS
SELECT SourceID, CAST(FIELDS AS TEXT) AS FieldsTxt
FROM SourceTable
WHERE CAST(FIELDS AS TEXT) LIKE '%<Field><Name>ForceFF</Name><Value>GEDCOM</Value></Field>%'
;
 
UPDATE OR REPLACE xTmpHybridSourcesUnSetFF
SET FieldsTxt =
REPLACE (
         FieldsTxt
         , '<Field><Name>ForceFF</Name><Value>GEDCOM</Value></Field>'
         , '<Field><Name>ForceFF</Name><Value/></Field>'
         )
;
 
UPDATE OR REPLACE SourceTable
SET FIELDS =
(
SELECT CAST(FieldsTxt AS BLOB) FROM xTmpHybridSourcesUnSetFF
 WHERE SourceTable.SourceID = xTmpHybridSourcesUnSetFF.SourceID
)
WHERE SourceID IN (SELECT SourceID FROM xTmpHybridSourcesUnSetFF)
;

Children – Set Lineage to Birth #birth #ancestors #lineage

RootsMagic 7 and prior has the peculiar shortcoming that ancestral lineage reports, charts and other important functions follow the lineage set by the parent family displayed in the last view of each person in one of the Main screens. If the last time you viewed your mother showed her stepfather as her father, then her birth father’s line is completely omitted while the stepfather’s line is included. This affects several reports and charts:

  • Ancestor narratives
  • Ahnentafel
  • Ancestor Box Chart
  • Pedigree Chart
  • Photo Tree
  • Ancestor Wall Chart

Moreover, it also affects the filter for selecting ancestors of the highlighted person in RootsMagic Explorer’s Mark/Unmark Group functions used for

  • other reports
  • GEDCOM export
  • colour coding
  • Named Groups

RootsMagic provides no option for any of these to be directed to follow the birth line despite having the feature in the Edit Person window to set the type of relationship between a child and a parent couple to other kinds such as adoptive or step. There seems to be little point to the Relationship setting because it is visible only in the Edit Person screen and is otherwise totally ignored. There has been a long outstanding Wish List item for the system to put the relationship setting to effective use.

Meanwhile, here is a potentially useful script if you have the typical family tree with several people having more than one parent couple and want to restore the focus to the birth line without having to manually search for those with other relationships to ensure that their Main View is set to the birth parents. It has been tested on a small trial database so, as always, be sure you have a backup before using it on your database or try it on a copy of your database. If you have the same database open in both RootsMagic and the SQLite manager from which you ran the script, you will not see any effect in RootsMagic until you refresh the Main Screen by switching to a different View.

Children-SetLineage2Birth.sql

-- Children-SetLineage2Birth.sql
/*
2015-10-18 Tom Holden ve3meo
 
Sets the displayed lineage to that of the Birth parents so that
lineage views and reports follow the birth line.
In the case of multiple birth parents (permitted by RootsMagic),
sets the birth family to the one with the highest Proof setting,
else to the lowest FamilyID.
 
It does not (or cannot?) cover all possible combinations that
a user might wish to allow or consider to be the preferred lineage
because a parent family (couple) may have only one member (mother
or father) and each spouse of a couple may have any of the four
proof settings for the birth relationship (0-undefined, 1-proven,
2-disproven, 3-disputed).
 
*/
 
DROP VIEW IF EXISTS BirthCouple
;
-- this view of the parents includes only birth parents in descending order of proof
CREATE TEMP VIEW BirthCouple AS
SELECT  ChildID, FamilyID FROM ChildTable
WHERE RelFather = 0 AND RelMother = 0  -- birth relationships
AND ProofFather < 2 AND ProofMother < 2  -- relationship proof is proven or undefined
ORDER BY ChildID, ProofFather DESC, ProofMother DESC -- priority to 'proven' relationships
;
 
-- set the display lineage to the birth parents with highest level of proof and
-- do not touch persons with no parents
UPDATE PersonTable
  SET ParentID =
    (SELECT FamilyID FROM BirthCouple BC
       WHERE PersonID = BC.ChildID)
  WHERE PersonID IN (SELECT ChildID FROM BirthCouple)
;
 
-- the temporary SQLite View BirthCouple will be deleted when the SQLite
-- manager closes the database

Ancestry.com and RootsMagic 5 #ancestrycom #sources #update #gedcom

Problems with GEDCOM

In my experience, building an evidence-driven Family Tree on Ancestry.com
from evidence found in its databases is faster and easier than extracting
the evidence from Ancestry.com and building the documented tree in
RootsMagic. But my RootsMagic database remains superior in other respects
and is my master database for reporting. So I have a number of problems
marrying my Ancestry and RootsMagic family trees and synchronizing them:

  • excessive exception report from the GEDCOM import
  • no images downloaded and linked
  • verbose footnotes

Avoid Glut of
Unknown Info _APID on Import

When you import into a RootsMagic database a GEDCOM downloaded from an
Ancestry Family Tree, RootsMagic generates an exception report file with the
.LST extension and with the same name and the same location as the database.
It is filled with a glut of exceptions of the form:

Unknown info (line 28)
    3 _APID 1,8944::212

These obscure the more interesting exceptions. Since RootsMagic does not
import these _APID lines, deleting them from the GEDCOM prior to import is
harmless while clarifying the LST resport. I use this regular expression
global search and replace in PSPad to clean them out:

Find: ^.+_APID.+$
Replace:

For the tree that is the subject of this page, it deleted 4461 occurrences
of the _APID tag from the GEDCOM.

Getting
Source and Other Images from Ancestry.com

The GEDCOM that you download from an Ancestry Family Tree does not contain
any images but it does include a URL to the Source page for each citation
that RootsMagic imports into its Citation Comments field. Conversely, Family
Tree Maker 2012 can synchronise with the Ancestry Family Tree and downloads
all the images; however, it is a bloated, slow program and does not include
the Ancestry source page URL in its export to GEDCOM 5.5 despite providing
the path to the local image file. Importing both GEDCOMs into a common
database and merging duplicate persons sounds plausible and easy but proved
useless. To re-marry the URLs
and the media items in a RootsMagic database, I developed the following
merging procedure which appears to be about 95% successful.

  1. Import the GEDCOM from the Ancestry Family Tree (optionally cleaned of
    _APID tags) to a new RootsMagic database named “AncestryGED.rmgc” (or
    whatever suits you).
  2. Optionally, inspect the corresponding LST file and make any changes in
    the RM database that you feel are important. However, these may be lost
    as this database is secondary in the merging process that follows.
  3. Using Family Tree Maker 2012, create a FTM database by synchronising
    to the Ancestry Family Tree.
  4. Export from FTM a GEDCOM 5.5, choosing FTM 2012 as the output format.
  5. Import the FTM GEDCOM into a new RootsMagic database named to suit
    you.
  6. Optionally, inspect the corresponding LST file for exceptions and make
    possible corrections in the RM database. This will be the primary
    database in the merge so these corrections should be preserved.
  7. Open SQLiteSpy with the fake RMNOCASE extension and open the primary
    database, the one created from importing the FTM2012 GEDCOM.
  8. In SQLiteSpy open the following SQL file which you will have
    downloaded to your scripts folder. MergeAncestryURLsToFTM.sql
  9. Edit the ATTACH command line so that the path and filename corresponds
    to that of your secondary database, the one created by importing the
    GEDCOM downloaded from Ancestry.com.
  10. Execute the script. It is slow because of the many matches and string
    processing to be done and the final status report is misleading as it
    only reports the number of records converted from Name citations unused
    by RM to Personal ones.
  11. Inspect your primary database to see that your citations now include
    the URL to the Ancestry source page in the Detail Comments and that the
    media item, if it existed before, is still there.
  12. You may now proceed to Shrinking
    Verbose Master Sources from Ancestry.com

The media are all stored in one folder under the Documents > Family Tree
Maker folder (assuming a default installation). Should you relocate them,
you will need to use RM’s global Search and Replace on multimedia filenames
or its Repair Broken Media Links tool to restore connections. I’m unsure of
the implications for the longterm if you wish to continue terr-building and
synchronising on Ancestry. Will existing names be preserved? Note that a
typical name is 1891 England Census-14.jpg or
1900 United States Federal Census-2.jpg, i.e., the source name with
a serial suffix when there is more than one image from the master source.
I’m inclined to leave them where they are, names unchanged, in the hope that
FTM will download only new images and continue the serial numbers.

Shrinking
Verbose Master Sources from Ancestry.com

One problem is that citations from Ancestry are extremely verbose. So I
tackled this one first. Having examined both the GEDCOM from Ancestry and
the imported results in RootsMagic, I elected to make the revisions
post-import as that had a more obvious solution for the tools and knowledge
I have.
My strategy is to replace the RootsMagic Footnote field value with the
ShortFootnote field value and the latter with the Master Source name. The
Bibliography value remains untouched and provides the same detail as the
original Footnote. Below is a comparison of a revised Master Source with its
original. Note that both exhibit a truncation that is the fault of
Ancestry.com.

Master
Source imported from Ancestry.com

Footnote:
Ancestry.com,
1870 United States Federal Census (Online publication – Provo, UT, USA:
Ancestry.com Operations, Inc., 2009. Images reproduced by
FamilySearch..Original data – 1870 U.S. census, population schedules. NARA
microfilm publication M593, 1,761 rolls. Washington, D.C.: National
Archives and Recor)

Short
Footnote:
Ancestry.com,
1870 United States Federal Census

Bibliography:
Ancestry.com.
1870 United States Federal Census. Online publication – Provo, UT, USA:
Ancestry.com Operations, Inc., 2009. Images reproduced by
FamilySearch..Original data – 1870 U.S. census, population schedules. NARA
microfilm publication M593, 1,761 rolls. Washington, D.C.: National
Archives and Recor.

Repository:

Ancestry.ca

http://www.Ancestry.ca

Master
Source after SQLite procedure

Footnote:
Ancestry.com,
1870 United States Federal Census

Short
Footnote:
1870
United States Federal Census

Bibliography:
Ancestry.com.
1870 United States Federal Census. Online publication – Provo, UT, USA:
Ancestry.com Operations, Inc., 2009. Images reproduced by
FamilySearch..Original data – 1870 U.S. census, population schedules. NARA
microfilm publication M593, 1,761 rolls. Washington, D.C.: National
Archives and Recor.

Repository:

Ancestry.ca

http://www.Ancestry.ca

Procedure

This involves multiple steps and tools:

  • SQLiteSpy
  • PSPad or other text editor with sorting capability
  • MS Excel or, possibly, other spreadsheet program, using AncestrySourceConverter.xlsx

The steps comprise:

  1. 1. Download the Ancestry GEDCOM or export it from the synchronised
    Family Tree Maker database in GEDCOM for FTM16 format.
  2. 2. Import the GEDCOM to a new RM database.
  3. 3. Backup your database.
  4. 4. Make a copy of your database with a different name.
  5. 5. Open the new database with SQLiteSpy.
  6. 6. Run the following
    query: SELECT SourceID, Name,
    SUBSTR(CAST(Fields AS TEXT),41) AS Fields FROM SourceTable;
  7. 7. Select all results and copy to the clipboard (select a cell,
    Ctrl-A, Ctrl-C)
  8. 8. Paste into a new text file in PSPad
  9. 9. Sort and/or edit so that the headings are at the top and the
    results are contiguous with no intervening blank lines. Copy all.
  10. 10. Open AncestrySourceConverter.xslx with MS Excel.
  11. 11. Delete all from sheet Source.
  12. 12. Paste from clipboard into the 1st cell of sheet Source.
  13. 13. Copy all UPDATE queries from sheet SQL. Note that just the first
    100 queries are created. If what you pasted into sheet Source is more
    than 100 rows, you will either have to do the UPDATEs in multiple
    batches or extend the rows of formulae in sheets Extract, Replace and
    SQL.
  14. 14. Paste into a new SQL window in SQLiteSpy.
  15. 15. Check through to the end that syntax colouring is correct. If
    there was an apostrophe in the original source it will need to be
    escaped with a second so that it is not interpreted as a text delimiter.
  16. 16. Execute the batch of UPDATE queries. This may take some 20 seconds
    – I do not know why it is so slow.
  17. 17. Open both the unprocessed and revised RootsMagic databases with RM
    and inspect the Master Source list to satisfy yourself with the results
    and make any other edits required.

To abbreviate the Short Footnote more, I think requires manual editing of
each citation, most effectively done by converting the sources to the !MyFreeForm3 source template.

Images of Results

MergedURLs&Images-CitationScreen.PNGMergedURLs&Images-MediaScreen.PNG

Generate WebTags from
the Citations

This applies to RootsMagic 6 which first introduced WebTags. The Ancestry
GEDCOM passes the URL (hyperlink) of each online citation in the citation
Comments field. For convenient access to these online sources, it is
desirable to make a Citation WebTag with that URL. That’s what is described
in WebTags
– from Ancestry.com and FTM
. And then to make it even more convenient,
copy those citation WebTags to person WebTags using WebTags
– Consolidate
.

Revealing
Citations from Ancestry that Disappeared

We think that citations in support of a Name in an Ancestry tree disappear
on import to RootsMagic in certain cases. See Citations
Invisible Revealed
for further details and how to reveal them to
RootsMagic.

Discussions & comments from Wikispaces site


ve3meo

Comment: “Open SQLiteSpy with the fake
RMNOCASE extension”

ve3meo
03
September 2018 19:40:01
ve3meo Sep 28, 2015
A Mac user encountering an error “LIKE pattern too large” or
words to that effect wanted to know if there was a way to run
the script without the RMNOCASE collation (on his own build of
sqlite3). The answer is a qualified “maybe” and it will take
trial and error. Wherever there is a field that is compared to
a value and that field has been defined to use the RMNOCASE
collation, it is probable that an error will result.
It is possible in some cases to override the defined collation
with a COLLATE NOCASE suffix wherever the field is named in a
SQL SELECT statement. However, you cannot UPDATE such a field.
Fortunately, the script does not UPDATE a field using the
RMNOCASE collation so any collation error must stem from a
comparison.
However, looking into this question raised some others. A
field that is UPDATEd is CitationTable.Comments and a puzzling
thing was observed. The script assumes that it is a TEXT field
which is what is contained in a number of databases inspected
yet in the table definition it is supposed to be type BLOB.
SQLite3 does not enforce field types and it appears that
RootsMagic does not follow its own field spec. This may be a
red herring but it could give rise to unexpected results if
some databases or rows within a database were not
type-consistent with others.