Media Repair Queries #filenames #duplicates #media #links #rmnocase #delete #update

Importing and merging duplicates of persons, families, places and sources already in a database can give rise to unwanted proliferation of duplicate items in the Media Gallery and multiple links to the same media item from a person, family, event, place or source. RMGC_Properties – Query now reports the quantities of such duplicates. A problem database that I worked on had 80 duplicate file names and 160 duplicate links from persons, families, events, places or sources. One image had 42 links from the same Place. As the Media Gallery grows, it becomes increasingly difficult to notice duplicate image file names and much harder to identify and remove duplicate links.

The series of queries in this SQL file help to identify duplicate file names and duplicate links and demonstrates a repair for one case of how duplicate file names can arise and a repair for duplicate links, regardless of cause. The repair queries delete records and, unfortunately, require the use of a SQLite manager capable of faking a RMNOCASE collation sequence to update the associated indexes. The investigative queries can be run on any SQLite manager. CAUTION: untested with RM5 and may give unwanted results due to changes in the media tables.

MediaRepair.sql

Query

1. Lists duplicate media file names

MediaIDMediaFileMediaPath
86_East Dyberry Cemetery 1.JPGd:My DocumentsGenealogyGravestonesUSA – East Dyberry (Wayne Co)
93_East Dyberry Cemetery 1.JPGE:My DocumentsGenealogyGravestonesUSA – East Dyberry (Wayne Co)
48bailie-agnes,1885-jan-26-(b).jpge:My DocumentsGenealogySurgeonerBirths
251bailie-agnes,1885-jan-26-(b).jpgd:My DocumentsGenealogySurgeonerBirths

2. Creates a list of UPDATE commands to replace the MediaID in MediaLinkTable that points to the MultimediaTable record having the path to E drive with the MediaID of the same file on the D drive. Copy this list to another SQL edit page and run it. NB – this query was for the specific problem of this database: the same files were on two different drives having the same paths.

3. Query #2 produces a list of queries such as this:

command
UPDATE medialinktable SET MediaID=106 WHERE MediaID=107;
UPDATE medialinktable SET MediaID=108 WHERE MediaID=109;
UPDATE medialinktable SET MediaID=110 WHERE MediaID=111;

This list is copied into a SQL edit window of the SQLite manager and executed, sans the header “command”.

4. After UPDATing the MediaIDs in MediaLinkTable to the new MediaIDs, then run this query to delete the records with oldMediaIDs from MultimediaTable. REQUIRES SharpPlus SQLite Developer or other SQLite manager supporting a fake RMNOCASE collation.

5. Lists duplicate links in MediaLinkTable to media files in MultimediaTable

LinkIDDUPESLinkedTo
1381_East Dyberry Cemetery 1.JPG
1375_East Dyberry Cemetery 7.JPG
761bailie-agnes,1885-jan-26-(b).jpg
72341Ballyclare Town Hall.pcx

6. DELETEs duplicate links from MediaLinkTable. REQUIRES SharpPlus SQLite Developer, SQLiteSpy with extension, or other SQLite manager supporting a fake RMNOCASE collation.

County Check #county

RootsMagic 5 introduced a new feature that checks what you have entered in the Place of a fact/event against a database of counties stored in the file CountyCheckDB.dat. This file is partially human readable when opened in a text editor. What it reveals is that there are four Internet sites that it is set to call for maps and other information:

TagURLDescription
FSWIKIhttps://www.familysearch.org/learn/wiki/en/Family Search Wiki
NAHCBhttps://publications.newberry.org/ahcbp/Newberry Atlas of Historical County Boundaries (US)
WIKIhttp://en.wikipedia.org/wikiWikipedia
http://maps.familysearch.orgEngland Jurisdictions 1851

The first three appear to get parameters for a specific place that are appended to the URL and passed to the CountyCheck controls “Online Info” and “Online Map” to open these sites on a specific page. The last one is called without parameters for places in England.

RM5 Version Monitoring #database

The database documentation was done largely on RootsMagic version 5.0.0.3 and those released in the several days following. It’s unlikely that the database schema has changed as of the most-recent version. However, any changes may be detected by opening a database created by each version with a SQLite manager, then exporting either a query of the full sqlite_master table or the database schema (just the SQL field of that table wrapped in a transaction) and comparing the exports from each version using a text editor that has a compare function (e.g. Notepad++).

The page RM5 Database System Catalog displays the full sqlite_master table from 5.0.0.6.

For future comparisons, below is the schema from 5.0.0.6:

BEGIN TRANSACTION;
CREATE TABLE AddressLinkTable (LinkID INTEGER PRIMARY KEY, OwnerType INTEGER, AddressID INTEGER, OwnerID INTEGER, AddressNum INTEGER, Details TEXT );
CREATE TABLE AddressTable (AddressID INTEGER PRIMARY KEY, AddressType INTEGER, Name TEXT COLLATE RMNOCASE, Street1 TEXT, Street2 TEXT, City TEXT, State TEXT, Zip TEXT, Country TEXT, Phone1 TEXT, Phone2 TEXT, Fax TEXT, Email TEXT, URL TEXT, Latitude INTEGER, Longitude INTEGER, Note BLOB );
CREATE TABLE ChildTable (RecID INTEGER PRIMARY KEY, ChildID INTEGER, FamilyID INTEGER, RelFather INTEGER, RelMother INTEGER, ChildOrder INTEGER, IsPrivate INTEGER, ProofFather INTEGER, ProofMother INTEGER, Note BLOB );
CREATE TABLE CitationTable (CitationID INTEGER PRIMARY KEY, OwnerType INTEGER, SourceID INTEGER, OwnerID INTEGER, Quality TEXT, IsPrivate INTEGER, Comments BLOB, ActualText BLOB, RefNumber TEXT, Flags INTEGER, FIELDS BLOB );
CREATE TABLE ConfigTable (RecID INTEGER PRIMARY KEY, RecType INTEGER, Title TEXT, DataRec BLOB );
CREATE TABLE EventTable (EventID INTEGER PRIMARY KEY, EventType INTEGER, OwnerType INTEGER, OwnerID INTEGER, FamilyID INTEGER, PlaceID INTEGER, SiteID INTEGER, DATE TEXT, SortDate INTEGER, IsPrimary INTEGER, IsPrivate INTEGER, Proof INTEGER, STATUS INTEGER, EditDate FLOAT, Sentence BLOB, Details BLOB, Note BLOB );
CREATE TABLE ExclusionTable (RecID INTEGER PRIMARY KEY, ExclusionType INTEGER, ID1 INTEGER, ID2 INTEGER );
CREATE TABLE FactTypeTable (FactTypeID INTEGER PRIMARY KEY, OwnerType INTEGER, Name TEXT COLLATE RMNOCASE, Abbrev TEXT, GedcomTag TEXT, UseValue INTEGER, UseDate INTEGER, UsePlace INTEGER, Sentence BLOB, Flags INTEGER );
CREATE TABLE FamilyTable (FamilyID INTEGER PRIMARY KEY, FatherID INTEGER, MotherID INTEGER, ChildID INTEGER, HusbOrder INTEGER, WifeOrder INTEGER, IsPrivate INTEGER, Proof INTEGER, SpouseLabel INTEGER, FatherLabel INTEGER, MotherLabel INTEGER, Note BLOB );
CREATE TABLE GroupTable (RecID INTEGER PRIMARY KEY, GroupID INTEGER, StartID INTEGER, EndID INTEGER );
CREATE TABLE LabelTable (LabelID INTEGER PRIMARY KEY, LabelType INTEGER, LabelValue INTEGER, LabelName TEXT COLLATE RMNOCASE, Description TEXT );
CREATE TABLE LinkTable (LinkID INTEGER PRIMARY KEY, extSystem INTEGER, LinkType INTEGER, rmID INTEGER, extID TEXT, Modified INTEGER, extVersion TEXT, extDate FLOAT, STATUS INTEGER, Note BLOB );
CREATE TABLE MediaLinkTable (LinkID INTEGER PRIMARY KEY, MediaID INTEGER, OwnerType INTEGER, OwnerID INTEGER, IsPrimary INTEGER, Include1 INTEGER, Include2 INTEGER, Include3 INTEGER, Include4 INTEGER, SortOrder INTEGER, RectLeft INTEGER, RectTop INTEGER, RectRight INTEGER, RectBottom INTEGER, Note TEXT, Caption TEXT COLLATE RMNOCASE, RefNumber TEXT COLLATE RMNOCASE, DATE TEXT, SortDate INTEGER, Description BLOB );
CREATE TABLE MultimediaTable (MediaID INTEGER PRIMARY KEY, MediaType INTEGER, MediaPath TEXT, MediaFile TEXT COLLATE RMNOCASE, URL TEXT, Thumbnail BLOB , Caption TEXT COLLATE RMNOCASE, RefNumber TEXT COLLATE RMNOCASE, DATE TEXT, SortDate INTEGER, Description BLOB);
CREATE TABLE NameTable (NameID INTEGER PRIMARY KEY, OwnerID INTEGER, Surname TEXT COLLATE RMNOCASE, Given TEXT COLLATE RMNOCASE, Prefix TEXT COLLATE RMNOCASE, Suffix TEXT COLLATE RMNOCASE, Nickname TEXT COLLATE RMNOCASE, NameType INTEGER, DATE TEXT, SortDate INTEGER, IsPrimary INTEGER, IsPrivate INTEGER, Proof INTEGER, EditDate FLOAT, Sentence BLOB, Note BLOB, BirthYear INTEGER, DeathYear INTEGER );
CREATE TABLE PersonTable (PersonID INTEGER PRIMARY KEY, UniqueID TEXT, Sex INTEGER, EditDate FLOAT, ParentID INTEGER, SpouseID INTEGER, Color INTEGER, Relate1 INTEGER, Relate2 INTEGER, Flags INTEGER, Living INTEGER, IsPrivate INTEGER, Proof INTEGER, Bookmark INTEGER, Note BLOB );
CREATE TABLE PlaceTable (PlaceID INTEGER PRIMARY KEY, PlaceType INTEGER, Name TEXT COLLATE RMNOCASE, Abbrev TEXT, Normalized TEXT, Latitude INTEGER, Longitude INTEGER, LatLongExact INTEGER, MasterID INTEGER, Note BLOB );
CREATE TABLE ResearchItemTable (ItemID INTEGER PRIMARY KEY, LogID INTEGER, DATE TEXT, SortDate INTEGER, RefNumber TEXT, Repository TEXT, Goal TEXT, SOURCE TEXT, RESULT TEXT );
CREATE TABLE RoleTable (RoleID INTEGER PRIMARY KEY, RoleName TEXT COLLATE RMNOCASE, EventType INTEGER, RoleType INTEGER, Sentence TEXT );
CREATE TABLE SourceTable (SourceID INTEGER PRIMARY KEY, Name TEXT COLLATE RMNOCASE, RefNumber TEXT, ActualText TEXT, Comments TEXT, IsPrivate INTEGER, TemplateID INTEGER, FIELDS BLOB );
CREATE TABLE SourceTemplateTable (TemplateID INTEGER PRIMARY KEY, Name TEXT COLLATE RMNOCASE, Description TEXT, Favorite INTEGER, Category TEXT, Footnote TEXT, ShortFootnote TEXT, Bibliography TEXT, FieldDefs BLOB );
CREATE TABLE WitnessTable (WitnessID INTEGER PRIMARY KEY, EventID INTEGER, PersonID INTEGER, WitnessOrder INTEGER, ROLE INTEGER, Sentence TEXT, Note BLOB, Given TEXT COLLATE RMNOCASE, Surname TEXT COLLATE RMNOCASE, Prefix TEXT COLLATE RMNOCASE, Suffix TEXT COLLATE RMNOCASE );
CREATE INDEX idxAddressName ON AddressTable (Name);
CREATE INDEX idxChildFamilyID ON ChildTable (FamilyID);
CREATE INDEX idxChildID ON ChildTable (ChildID);
CREATE INDEX idxChildOrder ON ChildTable (ChildOrder);
CREATE INDEX idxCitationOwnerID ON CitationTable (OwnerID);
CREATE INDEX idxCitationSourceID ON CitationTable (SourceID);
CREATE UNIQUE INDEX idxExclusionIndex ON ExclusionTable (ExclusionType, ID1, ID2);
CREATE INDEX idxFactTypeAbbrev ON FactTypeTable (Abbrev);
CREATE INDEX idxFactTypeGedcomTag ON FactTypeTable (GedcomTag);
CREATE INDEX idxFactTypeName ON FactTypeTable (Name);
CREATE INDEX idxFamilyFatherID ON FamilyTable (FatherID);
CREATE INDEX idxFamilyMotherID ON FamilyTable (MotherID);
CREATE INDEX idxGiven ON NameTable (Given);
CREATE INDEX idxLabelType ON LabelTable (LabelType);
CREATE INDEX idxLinkExtId ON LinkTable (extID);
CREATE INDEX idxLinkRmId ON LinkTable (rmID);
CREATE INDEX idxMediaCaption ON MediaLinkTable (Caption);
CREATE INDEX idxMediaFile ON MultimediaTable (MediaFile);
CREATE INDEX idxMediaOwnerID ON MediaLinkTable (OwnerID);
CREATE INDEX idxMediaURL ON MultimediaTable (URL);
CREATE INDEX idxNameOwnerID ON NameTable (OwnerID);
CREATE INDEX idxNamePrimary ON NameTable (IsPrimary);
CREATE INDEX idxOwnerDate ON EventTable (OwnerID,SortDate);
CREATE INDEX idxOwnerEvent ON EventTable (OwnerID,EventType);
CREATE INDEX idxPlaceAbbrev ON PlaceTable (Abbrev);
CREATE INDEX idxPlaceName ON PlaceTable (Name);
CREATE INDEX idxRecType ON ConfigTable (RecType);
CREATE INDEX idxResearchItemLogID ON ResearchItemTable (LogID);
CREATE INDEX idxResearchName ON ResearchTable (Name);
CREATE INDEX idxResearchOwnerID ON ResearchTable (OwnerID);
CREATE INDEX idxRoleEventType ON RoleTable (EventType);
CREATE INDEX idxSourceName ON SourceTable (Name);
CREATE INDEX idxSourceTemplateName ON SourceTemplateTable (Name);
CREATE INDEX idxSurname ON NameTable (Surname);
CREATE INDEX idxSurnameGiven ON NameTable (Surname, Given, BirthYear, DeathYear);
CREATE INDEX idxWitnessEventID ON WitnessTable (EventID);
CREATE INDEX idxWitnessPersonID ON WitnessTable (PersonID);
COMMIT TRANSACTION;

RM5 Table Summaries #datadefinitions

The following non-system tables currently reside in the RM5 database:

NameDescription
AddressLinkTableTranslates OwnerID from various tables to the key AddressID in AddressTable.
AddressTableStores Addresses for Persons in the tree, Repositories, and other entities that need not be in the tree. Note that the Name in this table is not the same as the names in the NameTable.
ChildTableStores relationship to family(ies) for each child. A child may have two or more families – e.g., birth and adoptive. Links to PersonTable for child and to FamilyTable for parents.
CitationTableStores Citation details with links to PersonTable, FamilyTable, and EventTable and to the SourceTable for the sources cited. One record per citation.
ConfigTableStores File settings, Report settings, Fonts, etc to preserve between sessions.
EventTableStores details for each Fact/Event with links to Persons, Families having the event and to the FactTypeTable for the Fact properties.
ExclusionTableStores Problem List items indicated by user as Not a problem.
FactTypeTableDefinition of the standard and user-defined Facts related to by the EventTable.
FamilyTableStores Family pairings, spousal order, and notes.
GroupTableStores data for Named Groups: ranges of PersonID’s from PersonTable that were marked for the group.
LabelTableStores labels for Named Groups and probably other labels, too.
LinkTableProbably has to do with New Family Search support and future support for remote Media.
MediaLinkTableStores relationship between records in MultiMediaTable and Person, Event, Place Tables along with settings for use in reports or scrapbooks, captions, notes (unused?), descriptions.
MultimediaTableStores path and name to Media files, a type of media code, URL (unused as of 5.0.0.6), and binary thumbnail for image files.
NameTableStores name parts for persons in tree, including Alternate Names as separate records (unlimited), primary and privacy settings, birth year and death year.
PersonTableStores basic data for persons in tree: globally Unique ID as well as internal PersonID, FamilyID of parent family and PersonID of spouse, calculated relationship to other person in tree, privacy and living flags, and general note.
PlaceTableStores system pre-defined and user-defined Places and Place Details (or sites) with Name, Abbreviation, Normalized Name, geographical coordinates, and notes. Linked from eventtable.
ResearchItemTableStores
ResearchTableStores contents of To-Do list with link to PersonTable or FamilyTable, depending on where task was created.
RoleTableStores system pre-defined and user-defined Roles with sentence templates for most FactTypes, linked from WitnessTable.
SourceTableStores Citation Source details for each unique source, linked from CitationTable (many to 1 allowed); linked to AddressTable for Source Repository and to SourceTemplateTable for support of Citation data entry and reports.
SourceTemplateTableStores system pre-defined and user-defined sentence and field templates for different kinds of citation sources, following published guidelines. Linked from SourceTable for support of Citation data entry and reports.
WitnessTableStores data for individual and family Event sharing among persons in tree and outside the tree, with links to RoleTable (Witness, Doctor, user-defined), EventTable, and to PersonTable (if in tree). Stores note for shared event plus name of person if not in tree. If person is not in tree, nothing of this record appears in reports.

RM5 Database System Catalog #datadefinitions

A singular system table, sqlite_master, resides in a SQLite database. The table in the RootsMagic 5 database essentially defines the RootsMagic 5 database tables, indices, and fields, as below.

Differences in structure with RootsMagic 4 include the addition of a new table, ResearchItemTable (and its index), to incorporate the newly-incorporated Research Log feature. In addition, MultimediaTable now includes five additional fields to account for the revised treatment of media throughout the program.

typenametbl_namerootpagesql
tableConfigTableConfigTable2CREATE TABLE ConfigTable (RecID INTEGER PRIMARY KEY, RecType INTEGER, Title TEXT, DataRec BLOB )
indexidxRecTypeConfigTable3CREATE INDEX idxRecType ON ConfigTable (RecType)
tablePersonTablePersonTable4CREATE TABLE PersonTable (PersonID INTEGER PRIMARY KEY, UniqueID TEXT, Sex INTEGER, EditDate FLOAT, ParentID INTEGER, SpouseID INTEGER, Color INTEGER, Relate1 INTEGER, Relate2 INTEGER, Flags INTEGER, Living INTEGER, IsPrivate INTEGER, Proof INTEGER, Bookmark INTEGER, Note BLOB )
tableFamilyTableFamilyTable5CREATE TABLE FamilyTable (FamilyID INTEGER PRIMARY KEY, FatherID INTEGER, MotherID INTEGER, ChildID INTEGER, HusbOrder INTEGER, WifeOrder INTEGER, IsPrivate INTEGER, Proof INTEGER, SpouseLabel INTEGER, FatherLabel INTEGER, MotherLabel INTEGER, Note BLOB )
indexidxFamilyFatherIDFamilyTable6CREATE INDEX idxFamilyFatherID ON FamilyTable (FatherID)
indexidxFamilyMotherIDFamilyTable8CREATE INDEX idxFamilyMotherID ON FamilyTable (MotherID)
tableChildTableChildTable10CREATE TABLE ChildTable (RecID INTEGER PRIMARY KEY, ChildID INTEGER, FamilyID INTEGER, RelFather INTEGER, RelMother INTEGER, ChildOrder INTEGER, IsPrivate INTEGER, ProofFather INTEGER, ProofMother INTEGER, Note BLOB )
indexidxChildIDChildTable11CREATE INDEX idxChildID ON ChildTable (ChildID)
indexidxChildFamilyIDChildTable12CREATE INDEX idxChildFamilyID ON ChildTable (FamilyID)
indexidxChildOrderChildTable13CREATE INDEX idxChildOrder ON ChildTable (ChildOrder)
tableEventTableEventTable14CREATE TABLE EventTable (EventID INTEGER PRIMARY KEY, EventType INTEGER, OwnerType INTEGER, OwnerID INTEGER, FamilyID INTEGER, PlaceID INTEGER, SiteID INTEGER, Date TEXT, SortDate INTEGER, IsPrimary INTEGER, IsPrivate INTEGER, Proof INTEGER, Status INTEGER, EditDate FLOAT, Sentence BLOB, Details BLOB, Note BLOB )
indexidxOwnerEventEventTable15CREATE INDEX idxOwnerEvent ON EventTable (OwnerID,EventType)
indexidxOwnerDateEventTable17CREATE INDEX idxOwnerDate ON EventTable (OwnerID,SortDate)
tableAddressTableAddressTable18CREATE TABLE AddressTable (AddressID INTEGER PRIMARY KEY, AddressType INTEGER, Name TEXT COLLATE RMNOCASE, Street1 TEXT, Street2 TEXT, City TEXT, State TEXT, Zip TEXT, Country TEXT, Phone1 TEXT, Phone2 TEXT, Fax TEXT, Email TEXT, URL TEXT, Latitude INTEGER, Longitude INTEGER, Note BLOB )
indexidxAddressNameAddressTable19CREATE INDEX idxAddressName ON AddressTable (Name)
tableFactTypeTableFactTypeTable20CREATE TABLE FactTypeTable (FactTypeID INTEGER PRIMARY KEY, OwnerType INTEGER, Name TEXT COLLATE RMNOCASE, Abbrev TEXT, GedcomTag TEXT, UseValue INTEGER, UseDate INTEGER, UsePlace INTEGER, Sentence BLOB, Flags INTEGER )
indexidxFactTypeNameFactTypeTable21CREATE INDEX idxFactTypeName ON FactTypeTable (Name)
indexidxFactTypeAbbrevFactTypeTable22CREATE INDEX idxFactTypeAbbrev ON FactTypeTable (Abbrev)
indexidxFactTypeGedcomTagFactTypeTable24CREATE INDEX idxFactTypeGedcomTag ON FactTypeTable (GedcomTag)
tableMultimediaTableMultimediaTable25CREATE TABLE MultimediaTable (MediaID INTEGER PRIMARY KEY, MediaType INTEGER, MediaPath TEXT, MediaFile TEXT COLLATE RMNOCASE, URL TEXT, Thumbnail BLOB , Caption TEXT COLLATE RMNOCASE, RefNumber TEXT COLLATE RMNOCASE, Date TEXT, SortDate INTEGER, Description BLOB)
indexidxMediaFileMultimediaTable26CREATE INDEX idxMediaFile ON MultimediaTable (MediaFile)
indexidxMediaURLMultimediaTable27CREATE INDEX idxMediaURL ON MultimediaTable (URL)
tableMediaLinkTableMediaLinkTable28CREATE TABLE MediaLinkTable (LinkID INTEGER PRIMARY KEY, MediaID INTEGER, OwnerType INTEGER, OwnerID INTEGER, IsPrimary INTEGER, Include1 INTEGER, Include2 INTEGER, Include3 INTEGER, Include4 INTEGER, SortOrder INTEGER, RectLeft INTEGER, RectTop INTEGER, RectRight INTEGER, RectBottom INTEGER, Note TEXT, Caption TEXT COLLATE RMNOCASE, RefNumber TEXT COLLATE RMNOCASE, Date TEXT, SortDate INTEGER, Description BLOB )
indexidxMediaOwnerIDMediaLinkTable30CREATE INDEX idxMediaOwnerID ON MediaLinkTable (OwnerID)
indexidxMediaCaptionMediaLinkTable31CREATE INDEX idxMediaCaption ON MediaLinkTable (Caption)
tableNameTableNameTable32CREATE TABLE NameTable (NameID INTEGER PRIMARY KEY, OwnerID INTEGER, Surname TEXT COLLATE RMNOCASE, Given TEXT COLLATE RMNOCASE, Prefix TEXT COLLATE RMNOCASE, Suffix TEXT COLLATE RMNOCASE, Nickname TEXT COLLATE RMNOCASE, NameType INTEGER, Date TEXT, SortDate INTEGER, IsPrimary INTEGER, IsPrivate INTEGER, Proof INTEGER, EditDate FLOAT, Sentence BLOB, Note BLOB, BirthYear INTEGER, DeathYear INTEGER )
indexidxNameOwnerIDNameTable34CREATE INDEX idxNameOwnerID ON NameTable (OwnerID)
indexidxSurnameNameTable35CREATE INDEX idxSurname ON NameTable (Surname)
indexidxGivenNameTable36CREATE INDEX idxGiven ON NameTable (Given)
indexidxSurnameGivenNameTable37CREATE INDEX idxSurnameGiven ON NameTable (Surname, Given, BirthYear, DeathYear)
indexidxNamePrimaryNameTable38CREATE INDEX idxNamePrimary ON NameTable (IsPrimary)
tablePlaceTablePlaceTable39CREATE TABLE PlaceTable (PlaceID INTEGER PRIMARY KEY, PlaceType INTEGER, Name TEXT COLLATE RMNOCASE, Abbrev TEXT, Normalized TEXT, Latitude INTEGER, Longitude INTEGER, LatLongExact INTEGER, MasterID INTEGER, Note BLOB )
indexidxPlaceNamePlaceTable41CREATE INDEX idxPlaceName ON PlaceTable (Name)
indexidxPlaceAbbrevPlaceTable42CREATE INDEX idxPlaceAbbrev ON PlaceTable (Abbrev)
tableResearchTableResearchTable43CREATE TABLE ResearchTable (TaskID INTEGER PRIMARY KEY, TaskType INTEGER, OwnerID INTEGER, OwnerType INTEGER, RefNumber TEXT, Name TEXT COLLATE RMNOCASE, Status INTEGER, Priority INTEGER, Date1 TEXT, Date2 TEXT, Date3 TEXT, SortDate1 INTEGER, SortDate2 INTEGER, SortDate3 INTEGER, Filename TEXT, Details BLOB )
indexidxResearchOwnerIDResearchTable44CREATE INDEX idxResearchOwnerID ON ResearchTable (OwnerID)
indexidxResearchNameResearchTable45CREATE INDEX idxResearchName ON ResearchTable (Name)
tableSourceTableSourceTable46CREATE TABLE SourceTable (SourceID INTEGER PRIMARY KEY, Name TEXT COLLATE RMNOCASE, RefNumber TEXT, ActualText TEXT, Comments TEXT, IsPrivate INTEGER, TemplateID INTEGER, Fields BLOB )
indexidxSourceNameSourceTable48CREATE INDEX idxSourceName ON SourceTable (Name)
tableCitationTableCitationTable49CREATE TABLE CitationTable (CitationID INTEGER PRIMARY KEY, OwnerType INTEGER, SourceID INTEGER, OwnerID INTEGER, Quality TEXT, IsPrivate INTEGER, Comments BLOB, ActualText BLOB, RefNumber TEXT, Flags INTEGER, Fields BLOB )
indexidxCitationSourceIDCitationTable50CREATE INDEX idxCitationSourceID ON CitationTable (SourceID)
indexidxCitationOwnerIDCitationTable51CREATE INDEX idxCitationOwnerID ON CitationTable (OwnerID)
tableAddressLinkTableAddressLinkTable52CREATE TABLE AddressLinkTable (LinkID INTEGER PRIMARY KEY, OwnerType INTEGER, AddressID INTEGER, OwnerID INTEGER, AddressNum INTEGER, Details TEXT )
tableWitnessTableWitnessTable53CREATE TABLE WitnessTable (WitnessID INTEGER PRIMARY KEY, EventID INTEGER, PersonID INTEGER, WitnessOrder INTEGER, Role INTEGER, Sentence TEXT, Note BLOB, Given TEXT COLLATE RMNOCASE, Surname TEXT COLLATE RMNOCASE, Prefix TEXT COLLATE RMNOCASE, Suffix TEXT COLLATE RMNOCASE )
indexidxWitnessEventIDWitnessTable55CREATE INDEX idxWitnessEventID ON WitnessTable (EventID)
indexidxWitnessPersonIDWitnessTable56CREATE INDEX idxWitnessPersonID ON WitnessTable (PersonID)
tableLinkTableLinkTable57CREATE TABLE LinkTable (LinkID INTEGER PRIMARY KEY, extSystem INTEGER, LinkType INTEGER, rmID INTEGER, extID TEXT, Modified INTEGER, extVersion TEXT, extDate FLOAT, Status INTEGER, Note BLOB )
indexidxLinkRmIdLinkTable58CREATE INDEX idxLinkRmId ON LinkTable (rmID)
indexidxLinkExtIdLinkTable59CREATE INDEX idxLinkExtId ON LinkTable (extID)
tableRoleTableRoleTable60CREATE TABLE RoleTable (RoleID INTEGER PRIMARY KEY, RoleName TEXT COLLATE RMNOCASE, EventType INTEGER, RoleType INTEGER, Sentence TEXT )
indexidxRoleEventTypeRoleTable62CREATE INDEX idxRoleEventType ON RoleTable (EventType)
tableGroupTableGroupTable63CREATE TABLE GroupTable (RecID INTEGER PRIMARY KEY, GroupID INTEGER, StartID INTEGER, EndID INTEGER )
tableExclusionTableExclusionTable64CREATE TABLE ExclusionTable (RecID INTEGER PRIMARY KEY, ExclusionType INTEGER, ID1 INTEGER, ID2 INTEGER )
indexidxExclusionIndexExclusionTable65CREATE UNIQUE INDEX idxExclusionIndex ON ExclusionTable (ExclusionType, ID1, ID2)
tableSourceTemplateTableSourceTemplateTable66CREATE TABLE SourceTemplateTable (TemplateID INTEGER PRIMARY KEY, Name TEXT COLLATE RMNOCASE, Description TEXT, Favorite INTEGER, Category TEXT, Footnote TEXT, ShortFootnote TEXT, Bibliography TEXT, FieldDefs BLOB )
indexidxSourceTemplateNameSourceTemplateTable67CREATE INDEX idxSourceTemplateName ON SourceTemplateTable (Name)
tableLabelTableLabelTable69CREATE TABLE LabelTable (LabelID INTEGER PRIMARY KEY, LabelType INTEGER, LabelValue INTEGER, LabelName TEXT COLLATE RMNOCASE, Description TEXT )
indexidxLabelTypeLabelTable70CREATE INDEX idxLabelType ON LabelTable (LabelType)
tableResearchItemTableResearchItemTable288CREATE TABLE ResearchItemTable (ItemID INTEGER PRIMARY KEY, LogID INTEGER, Date TEXT, SortDate INTEGER, RefNumber TEXT, Repository TEXT, Goal TEXT, Source TEXT, Result TEXT )
indexidxResearchItemLogIDResearchItemTable304CREATE INDEX idxResearchItemLogID ON ResearchItemTable (LogID)

Comparing Two RM Databases #compare #database #rmnocase

I wasn’t sure whether to post this page on the RootsMagic 4 wiki or the RootsMagic 5 wiki, but it really applies to both.

I’ve been considering the problem of data loss when using the RM Drag and Drop facility. Behind the scenes, Drag and Drop consists of a GEDCOM export from one RM database followed immediately by a GEDCOM import into another RM database. So the problem of the data loss really reduces down to identifying and fixing the causes of data loss in GEDCOM export and identifying and fixing the causes of data loss in GEDCOM import. I’m not sure that users such as us can do very much about fixing the causes, but I think we might be able to help with identifying the causes. And a constant question that’s never been answered to my satisfaction is, just what data might be lost in a Drag and Drop operation? What are all the actual possible causes, not just what are some of the rumored possible causes?

My idea is as follows. What if I could Drag and Drop my entire database into a new, empty database, and then run some kind of comparison utility that would compare my original database to the copy, and to do so at the SQL table and row level? Even if the Drag and Drop were perfect, I could imagine some trivial differences. But since Drag and Drop is not perfect, I would hope that those trivial differences that couldn’t be prevented wouldn’t obscure the real differences that would represent defects in the Drag and Drop process.

My initial concept was to try to write SQL that would accomplish the required comparison, but a little searching of the Internet revealed that there are already utilities available for download that seem to be more than capable of doing the required download. The utility that I settled on for initial testing may be found at
http://www.codeproject.com/KB/database/SQLiteCompareUtility.aspx and is called simply SQLite Compare. It seems pretty slick. It will run comparisons, identify which tables are different between the original and the copy, and then will allow you to drill down to see the differences. It allows you to include or not include BLOB fields in the comparison.

To that end and as a preliminary test, I made a copy of my database with the RM Copy function and ran SQLite Compare. Because this was a database level copy rather than a Drag and Drop, I was expecting no differences. But there were a few tables with differences, namely the Address Table, the Media Link Table, the Research Table, the Source Table, and the Source Template Table. All the rest of the tables compared just fine. Being a bit puzzled as to why any of the tables should have had any differences, I tried using the SQLite Compare capability to drill down to see the differences. I shouldn’t have been surprised by the results, but I was just a little bit surprised (shame on me). My attempt to drill down failed with the following dreaded error message.

ERROR: data comparison failed (SQLite error no such collation sequence: RMNOCASE)

So I’m wondering if anybody can think of a way to get around this problem? For the purposes of what I’m trying to accomplish, I really don’t need RMNOCASE nor just plain NOCASE nor any other kind of case processing. Is there anyway to turn off RMNOCASE for a whole database (a copy of a copy sort of thing, of course – not my real database) in order to be able to run SQLite Compare. The truth is that even if I tried to write my own compare utility, I would have to fight with the RMNOCASE problem all the way through.

Jerry

Added later in the day on 12/6/2011: I’m totally new to the SQLite Compare utility. Upon further review and upon playing with it further, I think I slightly misinterpreted the initial results. It was not saying that there were differences in the Address Table, the Media Link Table, etc. after the copy. Rather, it was saying that there were what it calls “comparison errors” after the copy. And when I drilled down to see what was going on, the RMNOCASE error messages provided the details of the “comparison errors”. So I’m going to have to think about this a little more and experiment a little more. Given that the tables listed above had RMNOCASE error messages, why didn’t other tables that use RMNOCASE collation also have RMNOCASE error messages rather than comparing as equal?

Discussions & comments from Wikispaces site


ve3meo

Faux RMNOCASE

ve3meo
07 December 2011 01:08:51

Jerry, what I have done successfully in the past without obvious damage was to rename a UTF8 collation in the paid license version of SQLite Developer to RMNOCASE. The good news with RM5 is the REINDEX utility so that even if the faux collation results in something unwanted, the indexes can be rebuilt with the right collation.

Tom


ve3meo

Try a NOCASE Copy of the Database

ve3meo
07 December 2011 03:26:29

This idea works in principle.

1. Get the full DDL of your RM4 (sub RM5 if that’s what you are working on) into a text editor.
2. Global replace RMNOCASE with NOCASE and save as a SQL file named, say, RM4_Create_as_DB3_NOCASE.sql
3. Create new empty DB3 database; load and run the modified DDL to create a NOCASE mirror of the RM4 structure
4. With your DB3 open, execute this:
ATTACH DATABASE ‘full URI to the RM4 file’ AS RM4;
5. Then run the following statement for each table (NameTable is the example):
INSERT INTO NameTable SELECT * FROM [RM4].NameTable;
6. Now you have all the data from your RM4 database in a DB3 with standard collations.
7. Do the same thing for each control and test database
8. Now run SQLite_Compare against the control DB3 and test DB3. It should work without tripping over the RMNOCASE collation.

Tom


ve3meo

ve3meo
07 December 2011 04:07:41

 

File Not Found

has step 5 expanded to all RM4 tables. According to romer, there is one additional table to be covered in RM5.


thejerrybryan

Thanks for the RMNOCASE Info

thejerrybryan
07 December 2011 04:22:19

I think I’ll try Tom’s free solution, but it may be several days before I have time to try it.

I’m actually on 5 now, and I’m aware of the extra table. I agree that moving forward most posts on the Wiki should be posted to 5 rather than 4 unless there is an issue specific to 4.

Jerry


mfseeker

RM Drag and Drop

mfseeker
25 January 2012 15:03:28

Jerry,

Have you made any further progress with your original project of comparing RM databases to evaluate the accuracy of RM’s GEDCOM out to GEDCOM in cycles?


thejerrybryan

thejerrybryan
26 January 2012 04:21:54

I put the project on hold when I realized that the SQLite Compare Utility wouldn’t be able to perform meaningful compares after a drag and drop or export/import. The problem is that most table rows will be “renumbered” after a drag and drop or export/import.

Convert Database to NOCASE

I do have another idea how to approach the project, but I haven’t had time to try it out.

Jerry


ve3meo

Comment: “…a way to get around this problem?” (no such collation sequence: RMNOCASE)

ve3meo
03 September 2018 20:10:25

ve3meo Dec 6, 2011

See my discussion notes:
a) $ for a SQLite manager that can fake a RMNOCASE
b) free solution copying the data for both the control and the test RM databases to DB3 files with same structure but NOCASE instead of RMNOCASE and comparing the DB3’s.


ve3meo

Comment: “SQLite Compare”

ve3meo
06 September 2018 18:26:45

ve3meo Dec 6, 2011

Very interesting find!

All Citations – Query #citations

CTE Versions for RM7 and RM8

Revised code by Pat Jones using common table expressions:
Version for RM7 and below  AllCitations-RM7.sql  tested 2021-01-20

Version for #RM8 and above  AllCitations-RM8.sql revised 2022-04-27 to include duplicate uses

Unlike the old version, this script does not appear to be able to display “headless citations” or “phantom sources or citations” as Null fields as in the screenshot below from the original script from 2010.

Old version for RM7 and below

AllCitations.sql revised 2022-04-26 to correct a SQLite error detected by later versions of SQLite 

Lists all citations in the database from which citations of non-existent sources (‘phantoms’) and citations for non-existent events or persons (‘headless’) can be found, along with other useful information such as all citations per source.

The query builds a temporary table with index which it then queries with a filter to suppress duplicate reports of citations against Alternate Names. While the query could be made without a temporary table, it would run exponentially more slowly with larger databases without invoking measures that are deprecated by SQLite documentation and incompatible with SQLite versions < 3.6.3. The intermediate table overcomes this problem and may be used for additional queries that will execute very quickly.

SQLiteSpy-AllCitations.png
Screenshot from SQLiteSpy

Discussions & comments from Wikispaces site


ve3meo

 

Slow query – possible improvement w/o temp table

 

ve3meo
05 February 2010 15:26:27

Parking this idea here for reference.

Problem: I’ve encountered really slow query speed on larger databases, usually when both NameTable.IsPrimary and OwnerID are constraints. When just OwnerID is the constraint, the sqlite query optimizer correctly chooses the idxNameTableOwnerID index and it runs fast. Add the IsPrimary constraint and it chooses the idxNameTableIsPrimary index and it runs very slow.

Possible Solutions:
1. Use the INDEXED BY clause to override the optimisier. However, use of this clause to tune performance is a “No-No”, according to the docs and not supported by sqlite < 3.6.3, e.g., by SQLiteman and DBTools DBManager.

2. Temporary table built w/o the IsPrimary constraint but including IsPrimary as a field. Query the temp table with the IsPrimary constraint. Works well.

3. Newest idea from the sqlite newsgroup: when querying with both constraints, make the IsPrimary a formula or expression to trick the query optimiser to ignore it.
SELECT * FROM NameTable … WHERE OwnerID=’x’ AND +IsPrimary=1;

Ref:”Igor Tandetnik” <itandetnik@mvps.org> wrote in message news:hkh4a0$mk8$1@ger.gmane.org…


ve3meo

 

ve3meo
05 February 2010 16:17:49

I meant to post this against the AllFacts4Persons query which is where I first ran into the speed problem. Having just tested the slight modification of the query below by adding the ‘+’ operator to the IsPrimary constraints improved the execution time using sqlite 3.6.22 by a factor of 3883, from 1130s to 0.291s!!!

[code]SELECT FactTypeTable.Name COLLATE NOCASE AS Fact, ‘Principal’ AS ‘Role
Type’, NameTable1.OwnerID AS RIN, NameTable1.Surname COLLATE NOCASE AS
Surname, NameTable1.Suffix COLLATE NOCASE AS Suffix, NameTable1.Prefix
COLLATE NOCASE AS Prefix, NameTable1.Given COLLATE NOCASE AS ‘Given Name’,
NameTable2.OwnerID AS ‘Sharer RIN’, NameTable2.Surname COLLATE NOCASE AS
‘Sharer Surname’, NameTable2.Suffix COLLATE NOCASE AS ‘Sharer Suffix’,
NameTable2.Prefix COLLATE NOCASE AS ‘Sharer Prefix’, NameTable2.Given
COLLATE NOCASE AS ‘Sharer Given Name’, COUNT(1) AS Count
FROM EventTable
INNER JOIN FactTypeTable ON EventTable.EventType = FactTypeTable.FactTypeID
INNER JOIN FamilyTable ON EventTable.OwnerID = FamilyTable.FamilyID
INNER JOIN NameTable AS NameTable1 ON FamilyTable.FatherID =
NameTable1.OwnerID
INNER JOIN NameTable AS NameTable2 ON FamilyTable.MotherID =
NameTable2.OwnerID
WHERE EventTable.OwnerType = 1 AND +NameTable1.IsPrimary = 1 AND
+NameTable2.IsPrimary = 1
GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12[/code]


Geraniums

 

Citations for non-existent events or persons (‘headless’)

Geraniums
05 July 2010 09:36:07

I have a large area of pink in the report.

Do the “CitID” and “CrcID” numbers mean anything? There are no surnames or given names – I assume that’s what’s meant by “headless”.

How can I find out where in the RootsMagic database that these can be fixed.

Thanks,
Debbie


ve3meo

 

ve3meo
05 July 2010 12:17:15

CitID is the internal reference number for a citation and SrcID is the same for a Master Source, both numbers hidden from the RootsMagic user.

If you are seeing pink, I think you may be using SQLiteSpy -that’s how it indicates a Null value. A Null value for both Surname and Given likely indicates a “headless” citation, unless a person was entered in the database without being assigned any names – I think you may have a few examples of that.

Headless citations cannot be fixed in the RootsMagic app except through a GEDCOM export/import.

Tom

Scope of Names in Queries and Sub-queries #msaccess #subqueries

I have been running into some subtle little problems with queries and sub-queries that suggest to me that I don’t have an adequate understanding of the scope of names within SQL. To that end, I created the following very trivial queries to test against both SQLite and MS Access.

--  Test query #1
SELECT Z.PersonID FROM
  (
   SELECT P.PersonID
     FROM PersonTable AS P
     WHERE P.PersonID <= 5
  ) AS Z
 
--  Test query #2
SELECT P.PersonID FROM
  (
   SELECT P.PersonID
     FROM PersonTable AS P
     WHERE P.PersonID <= 5
  ) AS Z
 
  --  Test query #3
SELECT PersonID FROM
  (
   SELECT P.PersonID
     FROM PersonTable AS P
     WHERE P.PersonID <= 5
  ) AS Z

All three queries use the identical sub-query. The only thing the sub-query does is to create a virtual table containing one column and up to five rows, where the column is the PersonID from the PersonTable and the rows are the PersonID values from 1 to 5. In my database, all five rows exist. The five rows may or may not all exist in any other database depending on whether or not the first five people that were entered into the database have been deleted or merged since they were first entered, or indeed whether as many as five people were ever entered into the database.

Query #1 works in both SQLite and MS Access (except that you have to delete the comment for MS Access). This is the expected behavior. It seems to be the case that SQL is always happy with one alias for a table in the sub-query (the alias is P in this case) and another alias for the same table outside the sub-query (the alias is Z in this case).

Query #2 fails in SQLite and works in MS Access (except that you have to delete the comment for MS Access). I think the query should fail in both cases, and I find the fact that it works in MS Access to be extremely disappointing. It appears that SQLite is treating the alias P as local to the sub-query but that MS Access is treating the alias P as global to the entire query, both inside and outside the sub-query. That seems to explain some rather strange looking errors I have seen from MS Access when I used the same alias both inside and outside a sub-query or when I used the same alias in two different and (I thought) independent sub-queries. It’s beginning to seem to me that to use MS Access safely, any table that is aliased multiple times in the same query will have to have a distinct alias each time, even though would seem that sensible scope rules would not really make distinct aliases necessary. Which is to say, it sounds like you can’t say that aliases in MS Access have to be “unique within the same scope but don’t have to be unique when they appear in different scopes” because MS Access seems to have only one scope – namely the whole query. I may not be completely correct about MS Access’s scope rules, but treating all scopes as global and programming accordingly seems to be the only thing that always works safely for me in the MS Access environment.

Query #3 works works in both SQLite and MS Access (except that you have to delete the comment for MS Access). I suppose that’s ok, but it makes me uncomfortable to write my code that way where the table that goes with a column is implicit, even though the table in question is a virtual table created with a sub-query. So I will write my query/sub-query combinations in the style of Query #1 rather than in the style of Query #3.

Jerry

Copy Fact to Group #update #namedgroup #events

Ever wanted to copy the same fact to a bunch of other people in your database? See the RootsMagic Forum thread Globally add fact to group for some background and an alternative method using GEDCOM. Here’s the braveheart method. See Copy RIN to REFN for a special variant of this query and a procedure that is applicable to both.

CopyFact2Group.sql – rev 2020-05-16 corrected LabelID to LabelValue for the GroupID

-- CopyFact2Group.sql
-- Tom Holden 5 Apr 2011
-- rev 2020-05-16 corrected LabelID to LabelValue for the GroupID
-- ALWAYS BACK UP YOUR DATABASE BEFORE RUNNING A QUERY THAT MODIFIES IT
-- Copies a fact from one person to all persons in a named group
--  - Media is NOT copied but Sources and Note are.
--  - Edit date is not modified from that of the original fact.
-- Requires GroupID and EventID values to be found and entered, each into
--  two queries. The GroupID is the LabelValue of the LabelName corresponding
--  to the name of the Group in the LabelTable table.
--  The EventID is easily found if it is the last fact entered - the EventID
--  of the last row in the table EventTable.
--
-- The first query adds a record to the EventTable for each person with
--  the same fact values as the record copied.
-- The second query adds an identical record to the CitationTable for each source for the
--  copied fact for each added fact. If n sources for the original fact, then the same n
--  sources for the fact are added to each person in the group.
--
-- This query copies the fact/event but not the Sources to the persons in the Named Group
-- CHANGE values of GroupID and EventID to your values
INSERT OR ROLLBACK INTO EventTable (EventType, OwnerType, OwnerID, FamilyID, PlaceID, SiteID, DATE, SortDate,
       IsPrimary, IsPrivate, Proof, STATUS, EditDate, Sentence, Details, Note)
  SELECT EventType, OwnerType, N.OwnerID, FamilyID, PlaceID, SiteID, DATE, SortDate,
       IsPrimary, IsPrivate, Proof, STATUS, EditDate, Sentence, Details, Note
   FROM (SELECT N.OwnerID FROM NameTable N, GroupTable WHERE N.OwnerID >=StartID AND N.OwnerID <= EndID AND GroupID=???) N,
       EventTable E WHERE EventID=???;
 
-- This query copies the Sources for the newly created facts.
-- CHANGE value of C.OwnerID to your EventID value and GroupID to your GroupID
INSERT OR ROLLBACK INTO CitationTable (OwnerType, SourceID, OwnerID, Quality, IsPrivate, Comments, ActualText, RefNumber, Flags, FIELDS)
  SELECT C.OwnerType, C.SourceID, E.EventID, C.Quality, C.IsPrivate, C.Comments, C.ActualText, C.RefNumber, C.Flags, C.FIELDS
    FROM CitationTable C,
     (SELECT EventID FROM EventTable
        WHERE EventID >
         (SELECT MAX(EventID) FROM EventTable) - (SELECT SUM(EndID-StartID+1) FROM GroupTable
           WHERE GroupID=???)) E
    WHERE C.OwnerID=??? AND C.OwnerType=2;

Adding Virtual Columns to an RM4 Table #msaccess #todolist #colorcoding

(This is for the MS Access section of the Wiki. I thought I was putting it there, but it didn’t seem to go there. I guess I’m still figuring out how to work the Wiki. Jerry)

I’m about to describe what I think is a useful approach to some reporting problems that are of interest to me. But I’ll say up front that I’m probably going to abandon this approach because I think I have a better idea moving forward. But in the meantime, I’ll report an what I’ve done so far. Well, I’m going to abandon this useful approach at least in part, but there is at least one case where I will probably continue with this approach.

There are some cases where I want to report on some additional information that is not presently in any of the RM4 tables. So the basic idea is to create a “real” table in MS Access and to think of the “real” MS Access table as if it were a “virtual” SQLite table in the RM4 database. No changes will be made on the SQLite side of the house. I place the additional data on which I want to report in the MS Access table, JOIN the MS Access table with an appropriate SQLite table in the RM4 database, and I can use MS access to report on the JOIN’ed data just fine.

Of course, from the point of view of MS Access, the roles of “real” and “virtual” are just the reverse – the MS Access table is “real” and RM4 SQLite table is “virtual”. And from the point of view of MS Access, you can JOIN a “real” MS Access table with a “virtual” SQLite table in the RM4 database just fine. The idea is to create an MS Access table to contain the additional data, and to be sure that the MS Access table contains a column that can be JOIN’ed with an appropriate RM4 SQLite table.

The first example is an MS Access table that includes a column that has the names of the RM4 colors. I want to be able to display the string “Green” rather than the number 9, for example. And I want to be able sort and filter on the name of the color rather than the number code for the color. I’m sure there is a simple and straightforward SQL way to achieve the required mapping between numbers and colors without having to make a new table, but making a new table is what I know how to do and it is very much in the spirit of how relational databases are usually designed.

To that end, I created the following MyColorTable and linked it to the RM4 PersonTable, with the required JOIN being between PersonTable.Color and MyColorTable.ColorID.

MyColorTable.jpg MyColorTableJOIN.jpg

The MyColorTable above is the example I plan to continue using moving forward. The following idea is the one for which I think I have a better approach moving forward. For some number of individuals – 17 individuals or 32 individuals or 103 individuals or whatever – I may have five “to do” tasks that I wish to accomplish for each individual. I wish to create a grid with 17 or 32 or 103 or whatever rows, and with five columns. I want to treat the grid as a checklist where I simply check off the appropriate cell for each individual and for each “to do” task when that task is completed for that individual. In my experience, traditional “to do” management software doesn’t work in this very simple and user friendly fashion, including “to do” management that’s available in RM4 itself and “to do” management available in GenSmarts.

So the obvious solution is to create a ToDoTable in MS Access with one column containing the RM4 Record Number for the individuals of interest and the other columns corresponding to the “to do” tasks to be accomplished. The only data that each cell in the “to do” columns has to contain is a 0 or a 1, or a blank or a 1. The 1 is my check mark denoting that the task is complete

This being a relational database application, I went at the problem slightly differently than having a row in my ToDo table for each individual of interest. I don’t really want to constantly be mucking around with entering the RM4 Record Numbers of the 17 or 32 or 103 individuals that are currently of interest. Rather, I want to automate that process. So I really created my ToDo table in MS Access with a row for every individual in my RM4 database. And to allow for future growth of my RM4 database without having to muck around with my ToDo table, I really, really created my ToDo table in MS Access with many more rows rows than there are currently people in my RM4 database. My RM4 database currently has about 58,000 individuals. So I created my ToDo table in MS Access with 100,000 rows. Row 1 of the RecNo column in the ToDo table contains 1, row 2 of the RecNo column contains 2, and so forth through row 100,000 of the RecNo column contains 100,000. The required JOIN is between ToDo.RecNo and PersonTable.PersonID. Some of the rows will never be used, but that’s fine. At the present time, I only have a column for one of my five “to do” tasks. I’ll add the other four “to do” tasks later.

MyColorTableJOIN2.jpg

Finally, I linked PersonTable to NameTable so I could display useful information about the individuals, and I filtered by color. So the standard RM4 color coding could be used to select individuals rather than having to muck around with lists of RM4 Record Numbers in my ToDo table, and the final result is as follows.

MyColorTableRPT1.jpg

I’m using “green people” as my filter, individuals color coded as green in RM4. The PrgraphOk task being complete means that for these particular individuals, I have painstakingly edited all fact notes so that paragraphing and white space looks exactly the way I wish for it to look in a narrative report. So in theory, I never have to perform this task again for those individuals. When I first created this MS Access query, the PrgraphOK column was completely blank. I manually replaced each blank with a 1 as I completed my editing and review of the way those individuals appeared in a narrative report. If I continue in this vein, I will gradually color code more people as green in RM4 to create my expanded “to do” list, and if I wish I can filter my MS Access query not to display the rows where PrgraphOk has already been set to 1.

The only thing I don’t like about this approach is that the information in the PrgraphOk column really isn’t stored in RM4, and I might like it to be stored there. There are number of creative places in RM4 where I could store the fact that I have completed my paragraphing review. And having done so, my query could create the PrgraphOk column by testing the data in RM4 rather than by having the data stored in my own ToDo table separate from RM4. I would still have to store the “I completed my paragraphing review” data in RM4 manually when I completed my paragraphing review, but it would keep the data in RM4.

Jerry

Discussions & comments from Wikispaces site


ve3meo

MS Access Section

ve3meo
26 June 2011 00:48:40

There is no section other than what may be implied by navigation guidance. You could edit other MS Access pages to include a link to your page. You could also create a MS Access box in the side menu with links to MS Access pages.

Tom


ve3meo

ve3meo
26 June 2011 12:59:07

Also, tagging the page can assist others searching for relevant stuff.


ve3meo

Comment: the basic idea is to create a “real” table in MS Access‍

ve3meo
03 September 2018 19:06:08

ve3meo Jun 26, 2011

One can add user defined tables in the RootsMagic 4 SQLite3 database using a SQLite manager. So far, I have seen no action from RM4 updates that causes such tables to be modified or deleted. That’s not to say that it could not happen. The nice thing is that these user tables are backed up and restored through RM4’s own procedures and always stay with the database in question. The downside is that they are not directly usable with multiple RM4 databases. But one could create a SQLIte database with MyColorTable and other common tables and an SQLite query can use it with any RM4 database by ATTACH.

Tom
thejerrybryan Jun 26, 2011

Thanks for all the very useful input. I’ve ultimately decided not to add “virtual” tables to RM4 by creating real tables in Access and joining them to RM4. I’ve also ultimately decided not to add any “real” tables to the RM4 database to use in my queries.

It’s been a very useful exercise to learn how to do both, and if it was just me I would probably do one or the other or both. But I would like to think I’m going to be a part of a larger community of users writing SQL against the RM4 database. As such, I would like to be able to write queries that I can post on this Wiki for others to use, and I would like for such queries to “just work”. But they won’t “just work” if they depend on additional tables that nobody else but me has created.

Again, much thanks for all the help.

Jerry

Inline comments


ve3meo

Comment: One can add user defined tables in th…

ve3meo
26 June 2011 13:05:33

One can add user defined tables in the RootsMagic 4 SQLite3 database using a SQLite manager. So far, I have seen no action from RM4 updates that causes such tables to be modified or deleted. That’s not to say that it could not happen. The nice thing is that these user tables are backed up and restored through RM4’s own procedures and always stay with the database in question. The downside is that they are not directly usable with multiple RM4 databases. But one could create a SQLIte database with MyColorTable and other common tables and an SQLite query can use it with any RM4 database by ATTACH.

Tom


thejerrybryan

thejerrybryan
26 June 2011 16:46:42

Thanks for all the very useful input. I’ve ultimately decided not to add “virtual” tables to RM4 by creating real tables in Access and joining them to RM4. I’ve also ultimately decided not to add any “real” tables to the RM4 database to use in my queries.

It’s been a very useful exercise to learn how to do both, and if it was just me I would probably do one or the other or both. But I would like to think I’m going to be a part of a larger community of users writing SQL against the RM4 database. As such, I would like to be able to write queries that I can post on this Wiki for others to use, and I would like for such queries to “just work”. But they won’t “just work” if they depend on additional tables that nobody else but me has created.

Again, much thanks for all the help.

Jerry