??ࡱ? > ?? ???? ? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ??_ ?( ` ? ? `
? / ? 0 ? ?0 ?D T i m e s N e w R o m a n ?b ?b D?b ??0h?b h?b ??0 ?D S y m b o l e w R o m a n ?b ?b D?b ??0h?b h?b ??0 ?D M o n o t y p e S o r t s ?b ?b D?b ??0h?b h?b ??0 0 ?D C o u r i e r N e w t s ?b ?b D?b ??0h?b h?b ??0 1 ?
? ? @ @@``?? ?
@ ?n ??? " d d @ ??? ?????? @@ `` ?? " ? ?` ? k ? % * & 3 % $ % ! " #
& ' ) S ?~ ? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? ? ? ? ? 0 ? ??? ??? n? @ ? ???? ? ?8 ? ? ? ? ?? ?g ? ?4 2 d 2 d t?b ??0l?b > $ ???????? p ? p p ? 0 ? < ?4 B d B d ?b ?b > $ ? ??? ?? ? - ?4 2 0 0 0 . Y u . D e m c h e n k o . T E R E N A ?? M u l t i l i n g u a l I s s u e s i n I n f o r m a t i o n R e t r i e v a l a n d R e s o u r c e D e s c r i p t i o n O ? ? = ??U ? ? ?S Multilingual Issues in Information Retrieval and Resource Description Overview ?d T + ' ' / + # + ? ?* Yuri Demchenko, TERENA
demchenko@terena.nl ? + + ? 8 A ? ? In this presentation ?
? ?V Multilingual Issues in TERENA Technical Programme
Multilinguality: trends and developments
Technical Issues/Background
Data presentation and resource description format
Standards Overview
Metadata and Cataloging
Recent Development in Subject Gateways and SE
Cross-language Information Retrieval
REIS/TAP Initiatives Multilinguality Framework
?b w E k / w E k / ? V ? 4 < ? ?= TERENA Multilingual Community and TERENA Technical Programme ?
> ? ?: TERENA has 43 members from 34 countries speaking 30 languages
Multilingual issues always were in the scope of TERENA Technical Program
WG-i18n - WG on Internationalisation issues
C3 Project on messaging transliteration tools
MAITS - initiated by WG-i18n
Multilingual E-Mail Agent Testing
Multilingual issues in Subject Gateways, Section 2.13 in SG Handbook
Multilingual Support in Internet/IT Applications. Information page - http://www.terena.nl/projects/multiling/
Liaison with STD bodies
CEN/TC304 Character set technology - http://www.stri.is/TC304/default.html
IETF ?L ? L P ? L P ? ? = E ? ?( Multilinguality: trends and developments ?
) ? ?' Storing, processing, presentation and exchange of information in many languages
Interactive (protocol based/negotiated ) applications and non-interactive (resource description and information presentation)
Multilingual Search and Retrieval
Multilingual Subject Gateways and Search Engines
CLIR testing at TREC
Data Resource Model and Multilinguality
One or Multiple languages
Data format
Metadata (not part of Data but part of Resource)
References, links
Professional Thesauri (Resource Context) - base for multiple languages and language unification
?? P ~ " F ( ? P ~ " F ( ? ?
( ? 6 > ? ? Internet Applications ?
? ?? None-interactive Application: Electronic Mail
Correct Message Composition and Rendering
Interactive applications
WWW: HTTP/HTML
http-equiv="Content-Type" Content="text/html; charset=euc-jp"
Content Negotiation Protocol
Media features, attributes
Direct and hop-by-hop communication
Operational Applications
XML/DOM
(Internationalised) DNS
LDAP and X.500 (Language Support ?) ?? . * ? ? D . * ? ? D ?~ ? ,
, ? ? 7 ? ? ?0 I18n and ML issues at IETF and other STD bodies ?
1 ? ?A IETF Architectural Model of Multilingual support in Internet Applications - RFC 2130
Language and Charset/Encoding tagging
Content negotiation framework (IETF/W3C)
Point-to-point vs hop-by-hop
Message based vs Interactive vs Streaming
Internationalised DNS (IDN) - Internationalised Domain Names
vs E-Mail (SMTP, IMAP)
vs Routing (Routing Policy Specification Language (RPSL))
vs Network Management (SNMP textual presentation)
vs Network Security (TLS and IPSec)
Content Encoding normalisation (IETF/Unicode)
LSD-2 - Large Scale Services Deployment
IMAP language extension ?~ U & * G > ? n U & * G > ? n ?? U ^
8 0 2 P ? * ? ?I IETF Architectural Model of Multilingual support in Internet Applications ? ?? User Interface
Presentation
Culture
Locale
Language
On-the-wire
Coded Character Set - Repertoire of ISO-10646
Character Encoding Scheme - UTF-8 (ml-text), US-ASCII (e-mail), ISO8859-1
Transfer Encoding Scheme (Base64, QP)
?Z %
? %
? ? ? ? > F ? ?( Content Negotiation Framework (IETF/W3C) ?
) ? ?? Content Negotiation covers three elements
Expressing the capabilities of the sender and the data resource to be transmitted
Expressing the capabilities of a receiver
A protocol by which capabilities are exchanged
Abstract framework for content negotiation
(Content) (Transmit.data) (Data document)
[Author]----->-----[Sender]----->-----[Receiver]----->-----[User]
Transparent Content Negotiation in HTTP - RFC 2295
Protocol-independent Content Negotiation Framework - RFC 2703
Non-message resource transfer
End-to-end vs hop-by-hop negotiation
Use of directory and resolution services
CC/PP exchange protocol based on HTTP Extension Framework (W3C)
Composite Capability/Preference Profile: A user side framework for content negotiation ?? * ? + ? q l @ W * ? + z q l @ W ? # ? ? + ? ? Charset and Language tagging ? ? MIME types (RFC 2045-2049)
text, img, audio, video
Charset = Character Set + Character Encoding Scheme
Transfer Encoding Scheme
base64
quoted-printable
Other media attributes and features (e.g., resolution, color, language, etc.)
Language
RFC 1766
ISO639-2
?~ e N e N ?, ! ? . ? : B ? ? WWW: HTTP/HTML ? ? ? HTTP header includes information about the type of the transferred information and the character encoding for text-based information:
http-equiv="Content-Type" Content="text/html; charset=euc-jp"
The Content-Language entity header field describes the natural language(s) of the intended audience for the enclosed document:
http-equiv="Content-Type" Content-Language=se
Character encoding information in the META information of the HTML document:
?X ? = ? - N D ?? ? , ? ~ , ? ( 8 ? ? XML: Character Set tagging ? ?? C h a r a c t e r i s a t o m i c u n i t o f t e x t
A l l I S O 1 0 6 4 6 c h a r a c t e r s + T A B , C R , L F
T h e m e c h a n i s m f o r E n c o d i n g c a n v a r y f o r d i f f e r e n t c h a r a c t e r s
A l l X M L p r o c e s s o r s m u s t a c c e p t U T F - 8 a n d U T F - 1 6
C h a r a c t e r E n c o d i n g d e c l a r a t i o n i n X M L d o c u m e n t s o r e n t i t i e s ( s e c t i o n 4 . 3 . 3 )
E n c o d i n g D e c l : : = S e n c o d i n g E q E n c N a m e | E n c N a m e ) < ? x m l e n c o d i n g + U T F - 8 ? > < ? x m l e n c o d i n g + E U C - J P ? >
D e f a u l t C h a r a c t e r S e t E n c o d i n g - U T F - 8 a n d U T F - 1 6
A u t o d e t e c t i o n o f C h a r a c t e r E n c o d i n g ?l ! ' m L ? V ! ' m L ? V ??
G
? ; C ? ? XML: Language tagging ?
? ?? L a n g u a g e i d e n t i f i c a t i o n ( s e c t i o n 2 . 1 2 )
L a b e l l i n g l a n g u a g e o f t h e w h o l e d o c u m e n t , e n t i t y o r i t e m
T a g f o r i d e n t i f i c a t i o n o f l a n g u a g e s
L a n g u a g e I D : : = L a n g c o d e ( - S u b c o d e )
L a n g c o d e : : = I S O 6 3 9 C o d e | I a n a C o d e | U s e r C o d e
E x a m p l e s :
< p x m l : l a n g = " e n " > T h e q u i c k b r o w n f o x j u m p s o v e r t h e l a z y d o g . < / p >
< p x m l : l a n g = " e n - G B " > W h a t c o l o u r i s i t ? < / p >
< p x m l : l a n g = " e n - U S " > W h a t c o l o r i s i t ? < / p >
< s p w h o = " F a u s t " d e s c = ' l e i s e ' x m l : l a n g = " d e " >
< l > H a b e n u n , a c h ! P h i l o s o p h i e , < / l >
< l > J u r i s t e r e i , u n d M e d i z i n < / l >
< l > u n d l e i d e r a u c h T h e o l o g i e < / l >
< l > d u r c h a u s s t u d i e r t m i t h e i ? e m B e m ? h ' n . < / l >
< / s p > ?z ' ] Z
B G ' ] Z
B F ? ?
?
&
$ ? @ H ? ? Unicode Technical Reports ?
? ? The Unicode Standard, Version 3.0 - Just published! - http://www.unicode.org/unicode/uni2book/u2.html
Unicode 2.0 test page http://www.terena.nl/projects/multiling/euroml/tests/test-ucspages1ucs.html
Multilingual European Subsets of ISO/IEC 10646-1 http://www.stri.is/TC304/p10_1998_05_30.pdf
Unicode technical Reports
UTR #15: Unicode Normalization Forms, Version 18.0 I-D by Martin Duerst
UTR #17: Character Encoding Model
UTR #16: UTF-EBCDIC
UTR #10: Unicode Collation Algorithm
UTR #7: Plane 14 Characters for Language Tags ?d g ? ? % / ? ? ?? B Q Z ^ ? ? & 7 ? ?; Language Definition in DC Metadata set - DC.Language Format ? ?? < m e t a n a m e = " D C . L a n g u a g e " c o n t e n t = " e n " >
< m e t a n a m e = " D C . L a n g u a g e " s c h e m e = " r f c 1 7 6 6 " c o n t e n t = " e n " >
< m e t a n a m e = " D C . L a n g u a g e " s c h e m e = " I S O 6 3 9 - 2 c o n t e n t = " e n g " >
< m e t a n a m e = " D C . L a n g u a g e s c h e m e = " r f c 1 7 6 6 c o n t e n t = " e n - U S " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " z h " >
< m e t a n a m e = " D C . L a n g u a g e " c o n t e n t = " j a " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " e s " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " g e r m a n " >
< m e t a n a m e = " D C . L a n g u a g e l a n g = " f r c o n t e n t = " a l l e m a n d " > ? ? ? ?? ? l ) ) *
? < D ? ?V Language Definition in DC Metadata set - Field content language labelling/attributing ? V ? ?8 A w o r k i n S p a n i s h m a y b e a s s i g n e d t h e f o l l o w i n g m e t a d a t a :
< m e t a n a m e = " D C . L a n g u a g e s c h e m e = " r f c 1 7 6 6 c o n t e n t = " e s " >
< m e t a n a m e = " D C . T i t l e "
l a n g = " e s "
c o n t e n t = " L a M e s a V e r d e y l a S i l l a R o j a " >
< m e t a n a m e = " D C . T i t l e "
l a n g = " e n "
c o n t e n t = " T h e G r e e n T a b l e a n d t h e R e d C h a i r " > ?x t 7 ? 9 @ ? ? DC in Multiple Languages ?
? ?? T h e r e f e r e n c e l a n g u a g e o f I n t l D C c o m m u n i t y i s E n g l i s h , h o w e v e r t h e s e m a n t i c s o d D C e l e m e n t s a r e i n p r i n c i p l e e x p r e s s e d e q u a l l y w e l l i n a n y m o d e r n l a n g u a g e
T h e v e r s i o n s o f D C e l e m e n t s i n v a r i o u s l a n g u a g e s s h o u l d s h a r e a s i n g l e n a m e s p a c e u s i n g t o k e n s t h a t l o o k l i k e E n g l i s h w o r d s b u t s t a n d f o r u n i v e r s a l e l e m e n t s - h t t p : / / p u r l . o r g / d c / e l e m e n t s / 1 . 1 /
D C i n M u l t i p l e L a n g u a g e s R e g i s t r y p r o j e c t - h t t p : / / p u r l . o r g / d c / g r o u p s / l a n g u a g e s . h t m
U s e s R D F s c h e m a s t o s h a r e m a c h i n e - r e a d a b l e t o k e n s f o r t r a n s l a t i o n o f D C t e r m s i n m u l t i p l e l a n g u a g e s ( 2 6 l a n g u a g e s t o d a t e )
L i n k a g e t o a n d f r o m c e n t r a l D C n a m e s p a c e s e r v e r
R e g i s t r y a s D i c t i o n a r y / T h e s a u r i - u s e I n t e r l i n g u a s t o l i n k d i f f e r e n t t r a n s l a t i o n s
F o r m a l r e c o g n i t i o n a n d s t a n d a r d i z a t i o n p r o c e d u r e ?6 ? ? 1 ? ? 1 ?L O [ ?
P ? ? G ? ?7 Document Description with Unqualified DC and RDF syntax ?
8 ? ?w
Das Erdbeben in Chili
Heinrich von Kleist
XML Encoding (Character set) declaration
UTF-8/UTF-16 as default encoding ?D , + ! + + ! ??
P " { ? . ; ? ?= Recent Developments in Subject Gateways, Indexing, Searching ? ?? N R E N s p r o j e c t s
S u b j e c t g a t e w a y s
C o m m e r c i a l S e a r c h E n g i n e s
M u l t i l i n g u a l T e x t R e t r i e v a l a n d P r o c e s s i n g
T U S T E P s y s t e m - u s i n g f u z z y m u l t i l i n g u a l s e a c h i n g
C r o s s - L a n g u a g e I n f o r m a t i o n R e t r i e v a l ( C L I R ) t e s t i n g a t T R E C - 7 / T R E C - 8 C o n f e r e n c e s b y N I S T
?6 e 4 Z e 4 Z ? ? [ ? * 9 ? ?% Multilingual Subject Gateway (DESIRE) ? ? Developing multilingual subject gateways (SOSIG as example)
SOSIG accept any languages evaluated for quality
Translation should be coherent and checked
Different language version should be equally well maintained
SOSIG Cataloguing rules
TITLE will be displayed in the first language
ALTERNATIVE TITLE in other languages
DESCRIPTION will mention different languages in which resource is available
URI of all language versions
Labeling URI language
Library standards for multilingual provision
NISO Z39.53 Language codes
USMARC Language codes ?Z < ? ? - 1 < ? ? - 1 ? ? k ? + : ? ?: Multilingual provision in popular Internet Search Engines ? ?? Multilingual SE
AltaVista - http://www.altavista.com/ - 28 languages
Documents indexed as is
Automatic translation - very simple and naive
Euroseek - http://www.euroseek.com/ - 30 languages
FAST Advanced Search - http://www.alltheweb.com - 31 languages
Google - http://www.google.com/ - 11 languages
Other sites that have dedicated national sites
interface language
language resources
no special language policy
Excite - 11 countries
Lycos - 23 countries ?~ 6 F ? 0 A + 6 F ? 0 A + ?b ' \ 7 ( ? ?
, ? ?4 TUSTEP TUebingen System of Text Processing Programs ? $ ? ?/ 1. File structure
2. Multilingual capabilities
3. Internal data presentation
4. Database publishing/output data presentation
5. CGI
6. Sample implementation
http://lddv.zdv.uni-tuebingen.de/cgi-bin/opac/zdvlit
Try entries like Smith or Meier or...
http://lddv.zdv.uni-tuebingen.de/cgi-bin/km/npquery
?Z ? 7 & 5 ? 7 & 5 ?? ? 0 ? A I ? ?D Cross-Language Information Retrieval (CLIR) testing at TREC-7/TREC-8 ?
E ? ?? TREC - Text REtrieval Conference - http://trec.nist.gov/
Cross-Language Information Retrieval (CLIR) technologies
Using Intermediary or Interlingual representation
Latent Semantic Indexing
Generalised Vector Space Model, etc.
Computer translation
Machine-readable bilingual dictionaries
MultilingualThesauri
Participants: ETH/Eurospider, IBM, Xerox, Cornell, New Mexico Univ, TNO, others
?Z r 2 > R Q r 2 > R Q ??
" ? 3 ? ?A REIS Project/Initiative Multilinguality framework - First attempt ? ?? M u l t i p l e l a n g u a g e i n d e x i n g
m u l t i p l e l a n g u a g e d o c u m e n t s / i n d e x e s
C r o s s - l a n g u a g e S e a r c h i n g
A u t o m a t i c Q u e r y f o r w a r d i n g b a s e d o n t h e s a u r i o r M L d i c t i o n a r y
U s i n g f u z z y m u l t i l i n g u a l s e a r c h i n g / m a t c h i n g
M u l t i l i n g u a l i n f o r m a t i o n r e t r i e v a l
A u t o m a t i c t r a n s l a t i o n ( i f r e q u e s t e d )
T r a n s l a t i o n R e q u e s t P r o t o c o l
I n t e r n a l D a t a / I n d e x e s p r e s e n t a t i o n
L a n g u a g e a n d C h a r a c t e r E n c o d i n g t a g g i n g
X M L a s i n t e r n a l p r e s e n t a t i o n o f d a t a a n d X M L l a n g u a g e a n d c h a r s e t t a g g i n g
T e x t / C h a r s e t n o r m a l i s a t i o n ( U n i c o d e o r T U S T E P - l i k e )
M e t a d a t a a n d R e s o u r c e D e s c r i p t i o n
D C . L a n g u a g e d e f i n i t i o n a n d X M L / R D F / D C L a n g u a g e t a g g i n g ?? $ l # B $ ? " 7 $ l # B $ ? " 7 / ? ? ? ? ? ? " ? ' ? ) ? , ? - ? / P ??? b ? ?? ? ` ? ? ??? ??? ?f ?? ? ??? ` ? ??? 3? ? ??? ??? ??? ?? ` ? ??? ___ ??? ??? ??? ??? ?> ??? " d d = ??? ?????? ?? ??? " d d ? = ??? ?????? ? ? ? ? u A ?4 ? d ?O ? " ?i ?n ??? " d d ? ??? ?????? @@ `` P ?P 4 O i ` ? p ?@ ? ?@
^
?V
? ) ?? ?( ?
? ?p
?
? ?H ? ? ? ? ? ?d ? ? ? ?
? ' W? ??
?
? ?Z ? $?v ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? x 8?? ? ? v
?T ? ? Click to edit Master title style ? ! ?
! ?:
?
? ?T ? ??v ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? Sf ? ? ? v
? ? ?R Click to edit Master text styles
Second Level
Third Level
Fourth Level
Fifth Level ? !
?
S ?,
?
?` ? ??v ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? v
?? ? ? ?* ?
? ? ? ? =44OOii ?
?
?` ? D?v ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ? _? ? ? v
?a ? ? * ? ? ? ? =44OOii ?I
? !
?` ? ??v ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ? !??? ? ? v
? ? ? Slide_* ?( ? ?$ ? ? =44OOii ?Z ?F ? 1 ? l Y
? $ ? ? ~ ?? ?~
? "
? ?N ? ? ? ? ? ? ???2 ? ? ? ? ? 1 ? l $ ?~
? #
? ?N ? ? ? ? ? ? ???2 ? ? ? ? ? 1 I l Y ? ?F ? ? ?
? ) ? ? ?c ?8
? %
s ? B? C{ D E?8 F?@ ? ? ???? ? ? ? ? ? ? ? ? ? ?? ? @? ? ? ? ? ??? F ? ? h ? ?=? Zhz? zF z? \ F 3? @ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?0
? &
s ? B? C? D E?4 F?< ? ? ???? ? ? ? ? ? ? ? ? ? ?? ? @? ? ? ? ?
??? ? i ? ? < ? ? ?? ? <?? #i ?? ? ? ? @ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? g ? 5 ?0
? '
s ? B- C? D E?4 F?< ? ? ???? ? ? ? ? ? ? ? ? ? ?? ? @? ? ? ? ?
?? o ? ? ? ? ? *l ? ? ,J? ? ? ?? ? Jz ? o @ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? A r n * ?
? (
? ?? BK Co D E?4 F?< ? ???? ? ? ? ? ?
?? ( % + ( J 2 7 J Q + E % n E Q 7 @ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?H
? ? ?0 ? ? @? ?h ? ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?" I n t e r n a t i o n a l . p o t ?& ? ? ? ?? 0 ? % ?f ?( ? ?4p? ~?p?
? ?^
?
? ?6 ? ? ? ? ? ? ?
? @ _? ?p
?
? ?H ? ? ? ? ? ?d ? ? ? ?
? ? _? ??
?
? ?Z ? ?3? ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ???? ? ? ?
?T ? ? Click to edit Master title style ? ! ?
! ??
?
? ?Z ? d4? ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? HZjG ? ? ?
?W ? ?# Click to edit Master subtitle style ? $ ?
$ ?
?
?` ? ?4? ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ? ???? ? ? ?
?[ ? ? * ? ? ? ? =44OOii ?
?
?` ? $5? ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ? ?S
? ? ? ?
?] ? ? * ? ? ? ? =44OOii ?9
?
?` ? ?5? ? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ? ???? ? ? ?
? ? ? Slide 2_* ?
? ?$ ? ? =44OOii ?H
? ? ?0 ? ? @? ?h ? ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ? ? 0 ? ? P ? ?N ?( ?
? ?
?
? ?T ? 7? ? jJ ? ? jJ ? ? ? ? ? ? ? ? ? ?? ,G ? ?
?
?q ? ? * ?
? ? ?
? ? ?
?
? ?T ? d7? ? jJ ? ? jJ ? ? ? ? ? ? ? ? ? ??l ?G ? ? ?
?s ? ? * ?
? ? ?
? ? ?p
?
? ?0 ? ? ? ??1 ? ? ? ??? ? ? ? ?:
?
? ?T ? ?7? ? ?g ? ֳ ? ?g ? ֳ ? ? ? ? ? ? ? ? ? LL? ? ? ?
? ? ?R Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level ? !
?
S ?
?
? ?Z ? $8? ? jJ ? ? jJ ? ? ? ? ? ? ? ? ? ? ,l ? ? ?
?q ? ? * ?
? ? ?
? ? ?
?
? ?Z ? ?8? ? jJ ? ? jJ ? ? ? ? ? ? ? ? ? ? l ?l ? ? v
?s ? ? * ?
? ? ?
? ? ?H
? ? ?0 ? ?b?f ? @? ? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? ? @ ? ?0 ?( ?
? ?H
? ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? ? 0 ?( ? ?? ?( ?
? ??
? # ?l ? D6? ? g ? ?? ? g ? ?? ? ? ? ? ? ? ? ? ? ? ? ? h?? ? ? ?
? ? ??
? # ?l ? ?6? ? g ? ?? ? g ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ?
??_ ? ? ?
? ? ?H
? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ` ? x ? ?( ? @?
? x ?l
? x C ? ? :? ? ? ? x 8?? ? ?
?
? ? ?l
? x C ? ? d:? ? ? ? Sf ? ? ? ?
? ? ?H
? x ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? p ? d ? ?( ?
? d ?l
? d C ? ? $;? ? ? ? x 8?? ? ?
?
? ? ?l
? d C ? ? ?;? ? ? ? Sf ? ? ? ?
? ? ?H
? d ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ? ? ? ? ?( ?
? ? ?l
? ? C ? ? =? ? ? ? x 8?? ? ?
?
? ? ?l
? ? C ? ? d=? ? ? ? Sf ? ? ? ?
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ? ? l ? ?( ? V
? l ?l
? l C ? ? T? ? ? x 8?? ? ?
? ? ?l
? l C ? ? ?? ? ? Sf ? ? ?
? ? ?H
? l ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ? ? p ? ?( ? @<, 8<,
? p ?l
? p C ? ? t ? ? ? x 8?? ? ?
?
? ? ?l
? p C ? ? 4? ? ? Sf ? ? ? ?
? ? ?H
? p ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? K ?C ? ? $? ? ?? ?( ? ?
? ? ??
? ? # ?l ? 4? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ? ? ? x 8?? ? ?
?
? ? ??
? ? # ?l ? ?? a ? ?? ? a ? ?? ? ? ? ? ? ? ? ? ? ? ? ? Sf - ? ? ?
? ? ? ?8 ? L ? ? d
? $? ? ?L ?d ?6 ?@ ? ? ? ? ?
? ? ? ? ? ? ? ??
? ?
?` ? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? ? ? ? ? ?
?B ? ? ? ? ?
?
?
? ?Z ? ę ? ? ? ? ? ??1 ? ? ? ? ? ? ? ? ? O ? ?
?r ? ?, Resolution Service /
Directory (content MD) ? - - ?
- ??
? ?
? ?Z ? Ĝ ? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
P
? d
?B ? ? ? ?? ?
?
?
?` ? ? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? ? h
* O
?[ ? ? Communication/Network ? ?
?x B
? ?
? ?H ? ? D ? ? ??o ? ? ? ? ? L $ L P
?x B
? ?
? ?H ? ? D ? ? ??o ? ? ? ? ? , , D
?x B
? ? ?
? ?H ? ? D ? ? ??o ? ? ? ? ? ? ? ? \
?x B
? ? ?
? ?H ? ? D ? ? ??o ? ? ? ? ? ? ? ? ,
?? ?@ ?
? ? $
? ? ?
? ? $ ??
? ?
? ?Z ? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
? ? $
?B ? ? ? ??
? ?
? ?Z ? ?? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
? ?
?B ? ? ? ??
? ?
? ?Z ? $? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
? ? ?
?B ? ? ? ?? ?
? ?
?` ? ?? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
? ? ?
?N ? ? Language ? ?
?
?
? ?
?` ? Ԙ ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? .
? ?
?u ? ? Presentation
Culture
Locale ?$ ? ?? ?
? ?
?` ? ?? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? :
? ? ?
?^ ? ? Content
Transfer
Agent ? ?
?? ?N ?
? ? $
? ? ? ? ? ? ? | ??
? ?
? ?Z ? $?"? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
? ? $
?B ? ? ? ??
? ?
? ?Z ? ??"? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
? ?
?B ? ? ? ??
? ?
? ?Z ? ?"? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
? ? ?
?B ? ? ? ?? ?
? ?
?` ? ??"? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ?
? ? ?
?N ? ? Language ? ?
?
?
? ?
?` ? d?"? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? .
? ?
?u ? ? Presentation
Culture
Locale ?$ ? ?? ?
? ?
?` ? $?"? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? :
? ? ?
?^ ? ? Content
Transfer
Agent ? ?
?? ?
? ?
?` ? ??"? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? 9 ? 3
?\ ? ? Communication Protocol ? ?
?? ?@ ? L ? ? ?
? #? ? L ? ? ? ??
? !?
? ?T ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? L ? ? ? ?? ?
? "?
?f ??? ? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? Y ? ? ?
?\ ? ? Resource ?& ?
?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ? ? ? ? ?( ? ? ?
? ? ?l
? ? C ? ? ? ? ? x 8?? ? ?
?
? ? ?l
? ? C ? ? t? ? ? ?? ? ? ?
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? 0? ? ? ?( ? o
? ? ?l
? ? C ? ? ?}? ? ? x 8?? ? ?
? ? ?l
? ? C ? ? 4~? ? ? Sf ? ? ?
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ? | ? ?( ? ??? ???
? | ?l
? | C ? ? t ? ? ? x 8?? ? ?
? ? ?l
? | C ? ? ? ? ? ? Sf ? ? ?
? ? ?H
? | ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ? 4 ? ?( ? %
? 4 ?l
? 4 C ? ? 4
? ? ? x 8?? ? ?
? ? ?l
? 4 C ? ? ?
? ? ? Sf ? ? ?
? ? ?H
? 4 ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ? ? ? ?( ? ????
? ? ?l
? ? C ? ? ? ? ? ? x 8?? ? ?
? ? ?l
? ? C ? ? T|? ? ? / ?? ? ?
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? p? ? ? ?( ?
? ? ?l
? ? C ? ? $?"? ? ? x 8?? ? ?
"
? ? ?l
? ? C ? ? ??"? ? ? Sf ? ? ? "
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? @? , ? ?( ?
? , ?l
? , C ? ? ?~? ? ? x 8?? ? ?
? ? ?l
? , C ? ? ?~? ? ? Sf ? ? ?
? ? ?H
? , ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? P? ? ? ?( ? Xu@ ?@?
? ? ?l
? ? C ? ? 4?? ? ? x 8?? ? ?
?
? ? ?l
? ? C ? ? ??? ? ? Sf ? ? ?
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? `? t ? ?( ?
? t ?l
? t C ? ? ?? ? ? x 8?? ? ?
? ? ?l
? t C ? ? t?? ? ? Sf ? ? ?
? ? ?H
? t ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? `? ? ? ?( ?
? ? ?l
? ? C ? ? ??? ? ? x 8?? ? ?
?
? ? ?l
? ? C ? ? t?? ? ? Sf ? ? ? ?
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ?? L ? ?( ?
? L ?l
? L C ? ? ?? ? ? x 8?? ? ?
? ? ?l
? L C ? ? t?? ? ? Sf ? ? ?
? ? ?H
? L ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? p? < ? ?( ?
? < ?l
? < C ? ? ԃ? ? ? x 8?? ? ?
? ? ?l
? < C ? ? 4?? ? ? Sf ? ? ?
? ? ?H
? < ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ?? @ ? ?( ? w@ #@
? @ ?l
? @ C ? ? T?? ? ? x 8?? ? ?
? ? ?l
? @ C ? ? ??? ? ? Sf ? ? ?
? ? ?H
? @ ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ?? ? ? ?( ?
? ? ?l
? ? C ? ? ??? ? ? x 8?? ? ?
? ? ?l
? ? C ? ? ?? ? ? Sf ? ? ?
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ?? ? ? ?( ?
? ? ?l
? ? C ? ? Đ"? ? ? x 8?? ? ?
9
? ? ?l
? ? C ? ? $?"? ? ? Sf ? ? ? 9
? ? ?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ?
? ? ?? ?? ? ?( ? ? ? @?
? ?l
? C ? ? d?? ? ? x 8?? ? ?
? ? ?l
? C ? ? ??? ? ? ?? `? ? ?
? ? ?H
? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? ?? ? 0 ? ?z ? ? ? ? ?( ? ????
? ? ?R
? ? 3 ? ? ? ? ??? ? ? ? ??
? ? C ? ? ? ? ? ? ? LL? ? ? ?
? ? ? ?H
? ? ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? * 0 ? ?z ?? ? ? ?( ?
? ? ?R
? ? 3 ? ? ? ? ??? ? ? ??
? ? C ? ? ?? ? ? ? LL? ? ?
? ? ? ?H
? ? ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? + 0 ? ?z ?? ? ? ?( ?
? ? ?R
? ? 3 ? ? ? ? ??? ? ? ??
? ? C ? ? ??? ? ? ? LL? ? ?
? ? ? ?H
? ? ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? , 0 ? ?z @? ? ? ?( ? 8
? ? ?R
? ? 3 ? ? ? ? ??? ? ? ??
? ? C ? ? ??? ? ? ? LL? ? ?
? ? ? ?H
? ? ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? 3 0 ? ?z P? ? ?( ?
? ?R
? 3 ? ? ? ? ??? ? ? ??
? C ? ? ?? ? ? ? LL? ? ?
? ? ? ?H
? ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? 7 0 ? ?z ? 0 ? ?( ?
? 0 ?R
? 0 3 ? ? ? ? ??? ? ? ??
? 0 C ? ? $?? ? ? ? LL? ? ?
? ? ? ?H
? 0 ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? 8 0 ? ?z ?? 8 ? ?( ? ? ?
? 8 ?R
? 8 3 ? ? ? ? ??? ? ? ??
? 8 C ? ? d?? ? ? ? LL? ? ?
? ? ? ?H
? 8 ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? 9 0 ? ?z ? D ? ?( ?
? D ?R
? D 3 ? ? ? ? ??? ? ? ??
? D C ? ? ??? ? ? ? LL? ? ?
? ? ? ?H
? D ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? : 0 ? ?z ? H ? ?( ? ? ??? <?
? H ?R
? H 3 ? ? ? ? ??? ? ? ??
? H C ? ? ??? ? ? ? LL? ? ?
? ? ? ?H
? H ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? ?? ? ; 0 ? ?z 0? P ? ?( ? |
? P ?R
? P 3 ? ? ? ? ??? ? ? ??
? P C ? ? D?? ? ? ? LL? ? ?
? ? ? ?H
? P ? ?0 ? ? ?b?f ? @? ? ? ? ? ??? ??? ̙ 33? ??? ??? r? ` ?w ?~ ?_ ?p ? 0 x? ?? {? ? 0 ?? ?? y? ;? " C? & ? [?
? ?? ?? ?? ?? ?? k? ۵ 5? 4 ?? 6 ? ?? ?? ? ? ۦ ?? ;? ? ?? {? [? ? 3 ?
?? A b ?x` ?( ` ? ?? ? ? ? ? ? ? ? ?
? ? ? ? ? ?
D ? A4 Paper (210x297 mm) ??
? Times New Roman Symbol Monotype Sorts Courier New International.pot T Multilingual Issues in Information Retrieval and Resource Description Overview In this presentation > TERENA Multilingual Community and TERENA Technical Programme ) Multilinguality: trends and developments Internet Applications 1 I18n and ML issues at IETF and other STD bodies J IETF Architectural Model of Multilingual support in Internet Applications ) Content Negotiation Framework (IETF/W3C) Charset and Language tagging WWW: HTTP/HTML XML: Character Set tagging XML: Language tagging Unicode Technical Reports < Language Definition in DC Metadata set - DC.Language Format W Language Definition in DC Metadata set - Field content language labelling/attributing DC in Multiple Languages 8 Document Description with Unqualified DC and RDF syntax > Recent Developments in Subject Gateways, Indexing, Searching & Multilingual Subject Gateway (DESIRE) ; Multilingual provision in popular Internet Search Engines 5 TUSTEP TUebingen System of Text Processing Programs E Cross-Language Information Retrieval (CLIR) testing at TREC-7/TREC-8 B REIS Project/Initiative Multilinguality framework - First attempt D Multilinguality Framework for Multilingual Indexing/Search Services Fonts Used Design Template
Slide Titles ? 6 >
_PID_GUID ? A N { 6 5 8 1 D F A 0 - 2 1 1 F - 1 1 D 4 - 9 B A B - 0 0 1 0 4 B 4 5 7 D F B } ?& _??? ? ? g?Yuri Demchenko D?b ??0h?b h?b ??0 ?D S y m b o l e w R o m a n ?b ?b D?b ??0h?b h?b ??0 ?D M o n o t y p e S o r t s ?b ?b D?b ??0h?b h?b ??0 0 ?D C o u r i e r N e w t s ?b ?b D?b ??0h?b h?b ??0 1 ?
? ? @ @@``?? ?
@ ?n ??? " d d @ ??? ?????? @@ `` ?? 2 ?* ?p ? m ? & * & 3 % $ % ! " #
& ' ) , S ?~ ? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? ? ? ? ? 0 ? ??? ??? n? @ ? ???? ? ?8 ? ? ? ? ?? ?g ? ?4 2 d 2 d t?b ??0l?b > $ ???????? p ? p p ? 0 ? < ?4 B d B d ?b ?b > $ ? ??? ?? ? - ?4 2 0 0 0 . Y u . D e m c h e n k o . T E R E N A ?? M u l t i l i n g u a l I s s u e s i n I n f o r m a t i o n R e t r i e v a l a n d R e s o u r c e D e s c r i p t i o n O ? ? = ?~V ? ? ?S Multilingual Issues in Information Retrieval and Resource Description Overview ?d T + ' ' / + # + ? ?* Yuri Demchenko, TERENA
demchenko@terena.nl ? + + ? 8 A ? ? In this presentation ?
? ?V Multilingual Issues in TERENA Technical Programme
Multilinguality: trends and developments
Technical Issues/Background
Data presentation and resource description format
Standards Overview
Metadata and Cataloging
Recent Development in Subject Gateways and SE
Cross-language Information Retrieval
REIS/TAP Initiatives Multilinguality Framework
?b w E k / w E k / ? V ? 4 < ? ?= TERENA Multilingual Community and TERENA Technical Programme ?
> ? ?: TERENA has 43 members from 34 countries speaking 30 languages
Multilingual issues always were in the scope of TERENA Technical Program
WG-i18n - WG on Internationalisation issues
C3 Project on messaging transliteration tools
MAITS - initiated by WG-i18n
Multilingual E-Mail Agent Testing
Multilingual issues in Subject Gateways, Section 2.13 in SG Handbook
Multilingual Support in Internet/IT Applications. Information page - http://www.terena.nl/projects/multiling/
Liaison with STD bodies
CEN/TC304 Character set technology - http://www.stri.is/TC304/default.html
IETF ?L ? L P ? L P ? ? = E ? ?( Multilinguality: trends and developments ?
) ? ?' Storing, processing, presentation and exchange of information in many languages
Interactive (protocol based/negotiated ) applications and non-interactive (resource description and information presentation)
Multilingual Search and Retrieval
Multilingual Subject Gateways and Search Engines
CLIR testing at TREC
Data Resource Model and Multilinguality
One or Multiple languages
Data format
Metadata (not part of Data but part of Resource)
References, links
Professional Thesauri (Resource Context) - base for multiple languages and language unification
?? P ~ " F ( ? P ~ " F ( ? ?
( ? 6 > ? ? Internet Applications ?
? ?? None-interactive Application: Electronic Mail
Correct Message Composition and Rendering
Interactive applications
WWW: HTTP/HTML
http-equiv="Content-Type" Content="text/html; charset=euc-jp"
Content Negotiation Protocol
Media features, attributes
Direct and hop-by-hop communication
Operational Applications
XML/DOM
(Internationalised) DNS
LDAP and X.500 (Language Support ?) ?? . * ? ? D . * ? ? D ?~ ? ,
, ? ? 7 ? ? ?0 I18n and ML issues at IETF and other STD bodies ?
1 ? ?A IETF Architectural Model of Multilingual support in Internet Applications - RFC 2130
Language and Charset/Encoding tagging
Content negotiation framework (IETF/W3C)
Point-to-point vs hop-by-hop
Message based vs Interactive vs Streaming
Internationalised DNS (IDN) - Internationalised Domain Names
vs E-Mail (SMTP, IMAP)
vs Routing (Routing Policy Specification Language (RPSL))
vs Network Management (SNMP textual presentation)
vs Network Security (TLS and IPSec)
Content Encoding normalisation (IETF/Unicode)
LSD-2 - Large Scale Services Deployment
IMAP language extension ?~ U & * G > ? n U & * G > ? n ?? U ^
8 0 2 P ? * ? ?I IETF Architectural Model of Multilingual support in Internet Applications ? ?? User Interface
Presentation
Culture
Locale
Language
On-the-wire
Coded Character Set - Repertoire of ISO-10646
Character Encoding Scheme - UTF-8 (ml-text), US-ASCII (e-mail), ISO8859-1
Transfer Encoding Scheme (Base64, QP)
?Z %
? %
? ? ? ? > F ? ?( Content Negotiation Framework (IETF/W3C) ?
) ? ?? Content Negotiation covers three elements
Expressing the capabilities of the sender and the data resource to be transmitted
Expressing the capabilities of a receiver
A protocol by which capabilities are exchanged
Abstract framework for content negotiation
(Content) (Transmit.data) (Data document)
[Author]----->-----[Sender]----->-----[Receiver]----->-----[User]
Transparent Content Negotiation in HTTP - RFC 2295
Protocol-independent Content Negotiation Framework - RFC 2703
Non-message resource transfer
End-to-end vs hop-by-hop negotiation
Use of directory and resolution services
CC/PP exchange protocol based on HTTP Extension Framework (W3C)
Composite Capability/Preference Profile: A user side framework for content negotiation ?? * ? + ? q l @ W * ? + z q l @ W ? # ? ? + ? ? Charset and Language tagging ? ? MIME types (RFC 2045-2049)
text, img, audio, video
Charset = Character Set + Character Encoding Scheme
Transfer Encoding Scheme
base64
quoted-printable
Other media attributes and features (e.g., resolution, color, language, etc.)
Language
RFC 1766
ISO639-2
?~ e N e N ?, ! ? . ? : B ? ? WWW: HTTP/HTML ? ? ? HTTP header includes information about the type of the transferred information and the character encoding for text-based information:
http-equiv="Content-Type" Content="text/html; charset=euc-jp"
The Content-Language entity header field describes the natural language(s) of the intended audience for the enclosed document:
http-equiv="Content-Type" Content-Language=se
Character encoding information in the META information of the HTML document:
?X ? = ? - N D ?? ? , ? ~ , ? ( 8 ? ? XML: Character Set tagging ? ?? C h a r a c t e r i s a t o m i c u n i t o f t e x t
A l l I S O 1 0 6 4 6 c h a r a c t e r s + T A B , C R , L F
T h e m e c h a n i s m f o r E n c o d i n g c a n v a r y f o r d i f f e r e n t c h a r a c t e r s
A l l X M L p r o c e s s o r s m u s t a c c e p t U T F - 8 a n d U T F - 1 6
C h a r a c t e r E n c o d i n g d e c l a r a t i o n i n X M L d o c u m e n t s o r e n t i t i e s ( s e c t i o n 4 . 3 . 3 )
E n c o d
! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i ? k l m ????o p q r s t u v w x y z { | } ~ ? ? ? ????????? ????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? n ????? ? ? ? ? j ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ????? ? R o o t E n t r y ch_html DEMCH_~2 1 D'Fe desire DESI ???????? ??d?O??? ? ?)? 1 & 1 ?/?ο? @ ex20C u r r e n t U s e r docs DOCS ?Ni
???E < @ >?? ?gga?I ?????????????H ??X?eQ? ?? ?9A ? . D?#R S u m m a r y I n f o r m a t i o n `? ??n?? ?F8??w? '( ???????? ui*? >???? P?p??? ??h?X?(?h?c
? w? P o w e r P o i n t D o c u m e n t ??G @H? d?}??U?-& G 5??( ????K?=.)??* ??? ?? ? ,?? ??9o??? ? ? ?? ?R?Q D o c u m e n t S u m m a r y I n f o r m a t i o n ? _????8 ???????????????r??? ?u???:Γu??7L? zf?fW :?lx? / ? ?4??KT?d????D?^?@?^4?֦I$?? ?? ??YAtc`
w RWw?????ݝ;:? r?A t?k ????????????=????T???? ???|??f2??3o m ??ǥ????q??L??.0?? n?5 ?߷ ?? ????????????? ?\?pe?cp???D??lynre*?P ??M?N?Q??a?(?IT `
? / ? 0 ? ?0 ?D T i m e s N e w R o m a n ?b ?b D?b ??0h?b h?b ??0 ?D S y m b o l e w R o m a n ?b ?b D?b ??0h?b h?b ??0 ?D M o n o t y p e S o r t s ?b ?b D?b ??0h?b h?b ??0 0 ?D C o u r i e r N e w t s ?b ?b D?b ??0h?b h?b ??0 1 ?
? ? @ @@``?? ?
@ ?n ??? " d d @ ??? ?????? @@ `` ?? * ?" ?h ? l ? & * & 3 % $ % ! " #
& ' ) , S ?~ ? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? ? ? ? ? 0 ? ??? ??? n? @ ? ???? ? ?8 ? ? ? ? ?? ?g ? ?4 2 d 2 d t?b ??0l?b > $ ???????? p ? p p ? 0 ? < ?4 B d B d ?b ?b > $ ? ??? ?? ? - ?4 2 0 0 0 . Y u . D e m c h e n k o . T E R E N A ?? M u l t i l i n g u a l I s s u e s i n I n f o r m a t i o n R e t r i e v a l a n d R e s o u r c e D e s c r i p t i o n O ? ? = ?mV ? ? ?S Multilingual Issues in Information Retrieval and Resource Description Overview ?d T + ' ' / + # + ? ?* Yuri Demchenko, TERENA
demchenko@terena.nl ? + + ? 8 A ? ? In this presentation ?
? ?V Multilingual Issues in TERENA Technical Programme
Multilinguality: trends and developments
Technical Issues/Background
Data presentation and resource description format
Standards Overview
Metadata and Cataloging
Recent Development in Subject Gateways and SE
Cross-language Information Retrieval
REIS/TAP Initiatives Multilinguality Framework
?b w E k / w E k / ? V ? 4 < ? ?= TERENA Multilingual Community and TERENA Technical Programme ?
> ? ?: TERENA has 43 members from 34 countries speaking 30 languages
Multilingual issues always were in the scope of TERENA Technical Program
WG-i18n - WG on Internationalisation issues
C3 Project on messaging transliteration tools
MAITS - initiated by WG-i18n
Multilingual E-Mail Agent Testing
Multilingual issues in Subject Gateways, Section 2.13 in SG Handbook
Multilingual Support in Internet/IT Applications. Information page - http://www.terena.nl/projects/multiling/
Liaison with STD bodies
CEN/TC304 Character set technology - http://www.stri.is/TC304/default.html
IETF ?L ? L P ? L P ? ? = E ? ?( Multilinguality: trends and developments ?
) ? ?' Storing, processing, presentation and exchange of information in many languages
Interactive (protocol based/negotiated ) applications and non-interactive (resource description and information presentation)
Multilingual Search and Retrieval
Multilingual Subject Gateways and Search Engines
CLIR testing at TREC
Data Resource Model and Multilinguality
One or Multiple languages
Data format
Metadata (not part of Data but part of Resource)
References, links
Professional Thesauri (Resource Context) - base for multiple languages and language unification
?? P ~ " F ( ? P ~ " F ( ? ?
( ? 6 > ? ? Internet Applications ?
? ?? None-interactive Application: Electronic Mail
Correct Message Composition and Rendering
Interactive applications
WWW: HTTP/HTML
http-equiv="Content-Type" Content="text/html; charset=euc-jp"
Content Negotiation Protocol
Media features, attributes
Direct and hop-by-hop communication
Operational Applications
XML/DOM
(Internationalised) DNS
LDAP and X.500 (Language Support ?) ?? . * ? ? D . * ? ? D ?~ ? ,
, ? ? 7 ? ? ?0 I18n and ML issues at IETF and other STD bodies ?
1 ? ?A IETF Architectural Model of Multilingual support in Internet Applications - RFC 2130
Language and Charset/Encoding tagging
Content negotiation framework (IETF/W3C)
Point-to-point vs hop-by-hop
Message based vs Interactive vs Streaming
Internationalised DNS (IDN) - Internationalised Domain Names
vs E-Mail (SMTP, IMAP)
vs Routing (Routing Policy Specification Language (RPSL))
vs Network Management (SNMP textual presentation)
vs Network Security (TLS and IPSec)
Content Encoding normalisation (IETF/Unicode)
LSD-2 - Large Scale Services Deployment
IMAP language extension ?~ U & * G > ? n U & * G > ? n ?? U ^
8 0 2 P ? * ? ?I IETF Architectural Model of Multilingual support in Internet Applications ? ?? User Interface
Presentation
Culture
Locale
Language
On-the-wire
Coded Character Set - Repertoire of ISO-10646
Character Encoding Scheme - UTF-8 (ml-text), US-ASCII (e-mail), ISO8859-1
Transfer Encoding Scheme (Base64, QP)
?Z %
? %
? ? ? ? > F ? ?( Content Negotiation Framework (IETF/W3C) ?
) ? ?? Content Negotiation covers three elements
Expressing the capabilities of the sender and the data resource to be transmitted
Expressing the capabilities of a receiver
A protocol by which capabilities are exchanged
Abstract framework for content negotiation
(Content) (Transmit.data) (Data document)
[Author]----->-----[Sender]----->-----[Receiver]----->-----[User]
Transparent Content Negotiation in HTTP - RFC 2295
Protocol-independent Content Negotiation Framework - RFC 2703
Non-message resource transfer
End-to-end vs hop-by-hop negotiation
Use of directory and resolution services
CC/PP exchange protocol based on HTTP Extension Framework (W3C)
Composite Capability/Preference Profile: A user side framework for content negotiation ?? * ? + ? q l @ W * ? + z q l @ W ? # ? ? + ? ? Charset and Language tagging ? ? MIME types (RFC 2045-2049)
text, img, audio, video
Charset = Character Set + Character Encoding Scheme
Transfer Encoding Scheme
base64
quoted-printable
Other media attributes and features (e.g., resolution, color, language, etc.)
Language
RFC 1766
ISO639-2
?~ e N e N ?, ! ? . ? : B ? ? WWW: HTTP/HTML ? ? ? HTTP header includes information about the type of the transferred information and the character encoding for text-based information:
http-equiv="Content-Type" Content="text/html; charset=euc-jp"
The Content-Language entity header field describes the natural language(s) of the intended audience for the enclosed document:
http-equiv="Content-Type" Content-Language=se
Character encoding information in the META information of the HTML document:
?X ? = ? - N D ?? ? , ? ~ , ? ( 8 ? ? XML: Character Set tagging ? ?? C h a r a c t e r i s a t o m i c u n i t o f t e x t
A l l I S O 1 0 6 4 6 c h a r a c t e r s + T A B , C R , L F
T h e m e c h a n i s m f o r E n c o d i n g c a n v a r y f o r d i f f e r e n t c h a r a c t e r s
A l l X M L p r o c e s s o r s m u s t a c c e p t U T F - 8 a n d U T F - 1 6
C h a r a c t e r E n c o d i n g d e c l a r a t i o n i n X M L d o c u m e n t s o r e n t i t i e s ( s e c t i o n 4 . 3 . 3 )
E n c o d i n g D e c l : : = S e n c o d i n g E q E n c N a m e | E n c N a m e ) < ? x m l e n c o d i n g + U T F - 8 ? > < ? x m l e n c o d i n g + E U C - J P ? >
D e f a u l t C h a r a c t e r S e t E n c o d i n g - U T F - 8 a n d U T F - 1 6
A u t o d e t e c t i o n o f C h a r a c t e r E n c o d i n g ?l ! ' m L ? V ! ' m L ? V ??
G
? ; C ? ? XML: Language tagging ?
? ?? L a n g u a g e i d e n t i f i c a t i o n ( s e c t i o n 2 . 1 2 )
L a b e l l i n g l a n g u a g e o f t h e w h o l e d o c u m e n t , e n t i t y o r i t e m
T a g f o r i d e n t i f i c a t i o n o f l a n g u a g e s
L a n g u a g e I D : : = L a n g c o d e ( - S u b c o d e )
L a n g c o d e : : = I S O 6 3 9 C o d e | I a n a C o d e | U s e r C o d e
E x a m p l e s :
< p x m l : l a n g = " e n " > T h e q u i c k b r o w n f o x j u m p s o v e r t h e l a z y d o g . < / p >
< p x m l : l a n g = " e n - G B " > W h a t c o l o u r i s i t ? < / p >
< p x m l : l a n g = " e n - U S " > W h a t c o l o r i s i t ? < / p >
< s p w h o = " F a u s t " d e s c = ' l e i s e ' x m l : l a n g = " d e " >
< l > H a b e n u n , a c h ! P h i l o s o p h i e , < / l >
< l > J u r i s t e r e i , u n d M e d i z i n < / l >
< l > u n d l e i d e r a u c h T h e o l o g i e < / l >
< l > d u r c h a u s s t u d i e r t m i t h e i ? e m B e m ? h ' n . < / l >
< / s p > ?z ' ] Z
B G ' ] Z
B F ? ?
?
&
$ ? @ H ? ? Unicode Technical Reports ?
? ? The Unicode Standard, Version 3.0 - Just published! - http://www.unicode.org/unicode/uni2book/u2.html
Unicode 2.0 test page http://www.terena.nl/projects/multiling/euroml/tests/test-ucspages1ucs.html
Multilingual European Subsets of ISO/IEC 10646-1 http://www.stri.is/TC304/p10_1998_05_30.pdf
Unicode technical Reports
UTR #15: Unicode Normalization Forms, Version 18.0 I-D by Martin Duerst
UTR #17: Character Encoding Model
UTR #16: UTF-EBCDIC
UTR #10: Unicode Collation Algorithm
UTR #7: Plane 14 Characters for Language Tags ?d g ? ? % / ? ? ?? B Q Z ^ ? ? & 7 ? ?; Language Definition in DC Metadata set - DC.Language Format ? ?? < m e t a n a m e = " D C . L a n g u a g e " c o n t e n t = " e n " >
< m e t a n a m e = " D C . L a n g u a g e " s c h e m e = " r f c 1 7 6 6 " c o n t e n t = " e n " >
< m e t a n a m e = " D C . L a n g u a g e " s c h e m e = " I S O 6 3 9 - 2 c o n t e n t = " e n g " >
< m e t a n a m e = " D C . L a n g u a g e s c h e m e = " r f c 1 7 6 6 c o n t e n t = " e n - U S " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " z h " >
< m e t a n a m e = " D C . L a n g u a g e " c o n t e n t = " j a " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " e s " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " g e r m a n " >
< m e t a n a m e = " D C . L a n g u a g e l a n g = " f r c o n t e n t = " a l l e m a n d " > ? ? ? ?? ? l ) ) *
? < D ? ?V Language Definition in DC Metadata set - Field content language labelling/attributing ? V ? ?8 A w o r k i n S p a n i s h m a y b e a s s i g n e d t h e f o l l o w i n g m e t a d a t a :
< m e t a n a m e = " D C . L a n g u a g e s c h e m e = " r f c 1 7 6 6 c o n t e n t = " e s " >
< m e t a n a m e = " D C . T i t l e "
l a n g = " e s "
c o n t e n t = " L a M e s a V e r d e y l a S i l l a R o j a " >
< m e t a n a m e = " D C . T i t l e "
l a n g = " e n "
c o n t e n t = " T h e G r e e n T a b l e a n d t h e R e d C h a i r " > ?x t 7 ? 9 @ ? ? DC in Multiple Languages ?
? ?? T h e r e f e r e n c e l a n g u a g e o f I n t l D C c o m m u n i t y i s E n g l i s h , h o w e v e r t h e s e m a n t i c s o d D C e l e m e n t s a r e i n p r i n c i p l e e x p r e s s e d e q u a l l y w e l l i n a n y m o d e r n l a n g u a g e
T h e v e r s i o n s o f D C e l e m e n t s i n v a r i o u s l a n g u a g e s s h o u l d s h a r e a s i n g l e n a m e s p a c e u s i n g t o k e n s t h a t l o o k l i k e E n g l i s h w o r d s b u t s t a n d f o r u n i v e r s a l e l e m e n t s - h t t p : / / p u r l . o r g / d c / e l e m e n t s / 1 . 1 /
D C i n M u l t i p l e L a n g u a g e s R e g i s t r y p r o j e c t - h t t p : / / p u r l . o r g / d c / g r o u p s / l a n g u a g e s . h t m
U s e s R D F s c h e m a s t o s h a r e m a c h i n e - r e a d a b l e t o k e n s f o r t r a n s l a t i o n o f D C t e r m s i n m u l t i p l e l a n g u a g e s ( 2 6 l a n g u a g e s t o d a t e )
L i n k a g e t o a n d f r o m c e n t r a l D C n a m e s p a c e s e r v e r
R e g i s t r y a s D i c t i o n a r y / T h e s a u r i - u s e I n t e r l i n g u a s t o l i n k d i f f e r e n t t r a n s l a t i o n s
F o r m a l r e c o g n i t i o n a n d s t a n d a r d i z a t i o n p r o c e d u r e ?6 ? ? 1 ? ? 1 ?L O [ ?
P ? ? G ? ?7 Document Description with Unqualified DC and RDF syntax ?
8 ? ?w
Das Erdbeben in Chili
Heinrich von Kleist
XML Encoding (Character set) declaration
UTF-8/UTF-16 as default encoding ?D , + ! + + ! ??
P " { ? . ; ? ?= Recent Developments in Subject Gateways, Indexing, Searching ? ?? N R E N s p r o j e c t s
S u b j e c t g a t e w a y s
C o m m e r c i a l S e a r c h E n g i n e s
M u l t i l i n g u a l T e x t R e t r i e v a l a n d P r o c e s s i n g
T U S T E P s y s t e m - u s i n g f u z z y m u l t i l i n g u a l s e a c h i n g
C r o s s - L a n g u a g e I n f o r m a t i o n R e t r i e v a l ( C L I R ) t e s t i n g a t T R E C - 7 / T R E C - 8 C o n f e r e n c e s b y N I S T
?6 e 4 Z e 4 Z ? ? [ ? * 9 ? ?% Multilingual Subject Gateway (DESIRE) ? ? Developing multilingual subject gateways (SOSIG as example)
SOSIG accept any languages evaluated for quality
Translation should be coherent and checked
Different language version should be equally well maintained
SOSIG Cataloguing rules
TITLE will be displayed in the first language
ALTERNATIVE TITLE in other languages
DESCRIPTION will mention different languages in which resource is available
URI of all language versions
Labeling URI language
Library standards for multilingual provision
NISO Z39.53 Language codes
USMARC Language codes ?Z < ? ? - 1 < ? ? - 1 ? ? k ? + : ? ?: Multilingual provision in popular Internet Search Engines ? ?? Multilingual SE
AltaVista - http://www.altavista.com/ - 28 languages
Documents indexed as is
Automatic translation - very simple and naive
Euroseek - http://www.euroseek.com/ - 30 languages
FAST Advanced Search - http://www.alltheweb.com - 31 languages
Google - http://www.google.com/ - 11 languages
Other sites that have dedicated national sites
interface language
language resources
no special language policy
Excite - 11 countries
Lycos - 23 countries ?~ 6 F ? 0 A + 6 F ? 0 A + ?b ' \ 7 ( ? ?
, ? ?4 TUSTEP TUebingen System of Text Processing Programs ? $ ? ?/ 1. File structure
2. Multilingual capabilities
3. Internal data presentation
4. Database publishing/output data presentation
5. CGI
6. Sample implementation
http://lddv.zdv.uni-tuebingen.de/cgi-bin/opac/zdvlit
Try entries like Smith or Meier or...
http://lddv.zdv.uni-tuebingen.de/cgi-bin/km/npquery
?Z ? 7 & 5 ? 7 & 5 ?? ? 0 ? A I ? ?D Cross-Language Information Retrieval (CLIR) testing at TREC-7/TREC-8 ?
E ? ?? TREC - Text REtrieval Conference - http://trec.nist.gov/
Cross-Language Information Retrieval (CLIR) technologies
Using Intermediary or Interlingual representation
Latent Semantic Indexing
Generalised Vector Space Model, etc.
Computer translation
Machine-readable bilingual dictionaries
MultilingualThesauri
Participants: ETH/Eurospider, IBM, Xerox, Cornell, New Mexico Univ, TNO, others
?Z r 2 > R Q r 2 > R Q ??
" ? 3 ? ?A REIS Project/Initiative Multilinguality framework - First attempt ? ?? M u l t i p l e l a n g u a g e i n d e x i n g
m u l t i p l e l a n g u a g e d o c u m e n t s / i n d e x e s
C r o s s - l a n g u a g e S e a r c h i n g
A u t o m a t i c Q u e r y f o r w a r d i n g b a s e d o n t h e s a u r i o r M L d i c t i o n a r y
U s i n g f u z z y m u l t i l i n g u a l s e a r c h i n g / m a t c h i n g
M u l t i l i n g u a l i n f o r m a t i o n r e t r i e v a l
A u t o m a t i c t r a n s l a t i o n ( i f r e q u e s t e d )
T r a n s l a t i o n R e q u e s t P r o t o c o l
I n t e r n a l D a t a / I n d e x e s p r e s e n t a t i o n
L a n g u a g e a n d C h a r a c t e r E n c o d i n g t a g g i n g
X M L a s i n t e r n a l p r e s e n t a t i o n o f d a t a a n d X M L l a n g u a g e a n d c h a r s e t t a g g i n g
T e x t / C h a r s e t n o r m a l i s a t i o n ( U n i c o d e o r T U S T E P - l i k e )
M e t a d a t a a n d R e s o u r c e D e s c r i p t i o n
D C . L a n g u a g e d e f i n i t i o n a n d X M L / R D F / D C L a n g u a g e t a g g i n g ?? $ l # B $ ? " 7 $ l # B $ ? " 7 ? B J ? ?2 REIS Project/Initiative Multilinguality framework ? 2 ? ? / ? ? ? ? ? ? " ? ' ? ) ? , ? - ? / P ??? b ? ?? ?
? y ?q ?? ? ? ?( ?
? ? ?l
? ? C ? ? ?"? ? ? x 8?? ? ?
"
? ? ?l
? ? C ? ? ??"? ? ? Sf ? ? ? "
? ? ?? ?
? ?
?` ? ę"? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? ??~
?Y ? ? To be developed yet ? ?
?H
? ? ? ?0 ? ? ? @? ?h ? ? ? ? ??? ??? ??? ?f ?? ? ??? r ?? B g4 ? J ?
?? 87 B b ??` ?( ` ? ? `
? / ? 0 ? ?0 ?D T i m e s N e w R o m a n ?b ?b ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ?& _????? ? f?Yuri Demchenko x ? ? ? ?
( 4 @
L X ` ?? HTTP и CGI TP = D:\msoffice\Templates\Presentation Designs\International.pot n Yuri Demchenko p 184 Microsoft PowerPoint 7.0 sen@ ???Tv @ ??Ԗѹ?@ `??X7?@ @???? ? G , ???? oZ ? . & ???? ? ? & ???? & # ???? TNPP ?? 0 D
v
&
TNPP ? & ???? & TNPP ??
? ??? - ? ??? " - - ! ? ?? - - ? " - - - - ! ? S?E - - - & ???? t ? ? & ???? - ? - ? ??? - $ t?t?v v ? ??? - ? $ v?v?x x ? ??? - ? $ x?x?{ { ? ??? - ? $ {?{?} } ? ??? - ? $ }?}? ? ??? - ? $ ??? ? ? ??? - ? $ ????? ? ? ??? - ? $ ????? ? ? ??? - ? $ ????? ? ? ??? - ? $ ????? ? ? ??? - ? $ ????? ? ? jj? - ? $ ????? ? ? UU? - ? $ ????? ? ? CC? - ? $ ????? ? ? ,,? - ? $ ????? ? ? ? - ? $ ????? ? - ? - - & ???? & ???? - - ZUO 7 - -
? - ??? ? Times New Roman - ? . $ 2
? ? Multilingual Issues 3 . ??? ? Times New Roman ? - ? .
2
? ? in . ??? ? Times New Roman ? - ? . . 2
? Information Retrieval and
) #
. . % 2
D? Resource Description#
& . - - 7 ?j - - ??? ? Times New Roman ? - ? . ( 2
?? Yuri Demchenko, TERENA . - - ? ? "System - ? & TNPP & ???? C ??
??՜.?? +,??D ??՜.?? +,??i n g D e c l : : = S e n c o d i n g E q E n c N a m e | E n c N a m e ) < ? x m l e n c o d i n g + U T F - 8 ? > < ? x m l e n c o d i n g + E U C - J P ? >
D e f a u l t C h a r a c t e r S e t E n c o d i n g - U T F - 8 a n d U T F - 1 6
A u t o d e t e c t i o n o f C h a r a c t e r E n c o d i n g ?l ! ' m L ? V ! ' m L ? V ??
G
? ; C ? ? XML: Language tagging ?
? ?? L a n g u a g e i d e n t i f i c a t i o n ( s e c t i o n 2 . 1 2 )
L a b e l l i n g l a n g u a g e o f t h e w h o l e d o c u m e n t , e n t i t y o r i t e m
T a g f o r i d e n t i f i c a t i o n o f l a n g u a g e s
L a n g u a g e I D : : = L a n g c o d e ( - S u b c o d e )
L a n g c o d e : : = I S O 6 3 9 C o d e | I a n a C o d e | U s e r C o d e
E x a m p l e s :
< p x m l : l a n g = " e n " > T h e q u i c k b r o w n f o x j u m p s o v e r t h e l a z y d o g . < / p >
< p x m l : l a n g = " e n - G B " > W h a t c o l o u r i s i t ? < / p >
< p x m l : l a n g = " e n - U S " > W h a t c o l o r i s i t ? < / p >
< s p w h o = " F a u s t " d e s c = ' l e i s e ' x m l : l a n g = " d e " >
< l > H a b e n u n , a c h ! P h i l o s o p h i e , < / l >
< l > J u r i s t e r e i , u n d M e d i z i n < / l >
< l > u n d l e i d e r a u c h T h e o l o g i e < / l >
< l > d u r c h a u s s t u d i e r t m i t h e i ? e m B e m ? h ' n . < / l >
< / s p > ?z ' ] Z
B G ' ] Z
B F ? ?
?
&
$ ? @ H ? ? Unicode Technical Reports ?
? ? The Unicode Standard, Version 3.0 - Just published! - http://www.unicode.org/unicode/uni2book/u2.html
Unicode 2.0 test page http://www.terena.nl/projects/multiling/euroml/tests/test-ucspages1ucs.html
Multilingual European Subsets of ISO/IEC 10646-1 http://www.stri.is/TC304/p10_1998_05_30.pdf
Unicode technical Reports
UTR #15: Unicode Normalization Forms, Version 18.0 I-D by Martin Duerst
UTR #17: Character Encoding Model
UTR #16: UTF-EBCDIC
UTR #10: Unicode Collation Algorithm
UTR #7: Plane 14 Characters for Language Tags ?d g ? ? % / ? ? ?? B Q Z ^ ? ? & 7 ? ?; Language Definition in DC Metadata set - DC.Language Format ? ?? < m e t a n a m e = " D C . L a n g u a g e " c o n t e n t = " e n " >
< m e t a n a m e = " D C . L a n g u a g e " s c h e m e = " r f c 1 7 6 6 " c o n t e n t = " e n " >
< m e t a n a m e = " D C . L a n g u a g e " s c h e m e = " I S O 6 3 9 - 2 c o n t e n t = " e n g " >
< m e t a n a m e = " D C . L a n g u a g e s c h e m e = " r f c 1 7 6 6 c o n t e n t = " e n - U S " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " z h " >
< m e t a n a m e = " D C . L a n g u a g e " c o n t e n t = " j a " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " e s " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " g e r m a n " >
< m e t a n a m e = " D C . L a n g u a g e l a n g = " f r c o n t e n t = " a l l e m a n d " > ? ? ? ?? ? l ) ) *
? < D ? ?V Language Definition in DC Metadata set - Field content language labelling/attributing ? V ? ?8 A w o r k i n S p a n i s h m a y b e a s s i g n e d t h e f o l l o w i n g m e t a d a t a :
< m e t a n a m e = " D C . L a n g u a g e s c h e m e = " r f c 1 7 6 6 c o n t e n t = " e s " >
< m e t a n a m e = " D C . T i t l e "
l a n g = " e s "
c o n t e n t = " L a M e s a V e r d e y l a S i l l a R o j a " >
< m e t a n a m e = " D C . T i t l e "
l a n g = " e n "
c o n t e n t = " T h e G r e e n T a b l e a n d t h e R e d C h a i r " > ?x t 7 ? 9 @ ? ? DC in Multiple Languages ?
? ?? T h e r e f e r e n c e l a n g u a g e o f I n t l D C c o m m u n i t y i s E n g l i s h , h o w e v e r t h e s e m a n t i c s o d D C e l e m e n t s a r e i n p r i n c i p l e e x p r e s s e d e q u a l l y w e l l i n a n y m o d e r n l a n g u a g e
T h e v e r s i o n s o f D C e l e m e n t s i n v a r i o u s l a n g u a g e s s h o u l d s h a r e a s i n g l e n a m e s p a c e u s i n g t o k e n s t h a t l o o k l i k e E n g l i s h w o r d s b u t s t a n d f o r u n i v e r s a l e l e m e n t s - h t t p : / / p u r l . o r g / d c / e l e m e n t s / 1 . 1 /
D C i n M u l t i p l e L a n g u a g e s R e g i s t r y p r o j e c t - h t t p : / / p u r l . o r g / d c / g r o u p s / l a n g u a g e s . h t m
U s e s R D F s c h e m a s t o s h a r e m a c h i n e - r e a d a b l e t o k e n s f o r t r a n s l a t i o n o f D C t e r m s i n m u l t i p l e l a n g u a g e s ( 2 6 l a n g u a g e s t o d a t e )
L i n k a g e t o a n d f r o m c e n t r a l D C n a m e s p a c e s e r v e r
R e g i s t r y a s D i c t i o n a r y / T h e s a u r i - u s e I n t e r l i n g u a s t o l i n k d i f f e r e n t t r a n s l a t i o n s
F o r m a l r e c o g n i t i o n a n d s t a n d a r d i z a t i o n p r o c e d u r e ?6 ? ? 1 ? ? 1 ?L O [ ?
P ? ? G ? ?7 Document Description with Unqualified DC and RDF syntax ?
8 ? ?w
Das Erdbeben in Chili
Heinrich von Kleist
XML Encoding (Character set) declaration
UTF-8/UTF-16 as default encoding ?D , + ! + + ! ??
P " { ? . ; ? ?= Recent Developments in Subject Gateways, Indexing, Searching ? ?? N R E N s p r o j e c t s
S u b j e c t g a t e w a y s
C o m m e r c i a l S e a r c h E n g i n e s
M u l t i l i n g u a l T e x t R e t r i e v a l a n d P r o c e s s i n g
T U S T E P s y s t e m - u s i n g f u z z y m u l t i l i n g u a l s e a c h i n g
C r o s s - L a n g u a g e I n f o r m a t i o n R e t r i e v a l ( C L I R ) t e s t i n g a t T R E C - 7 / T R E C - 8 C o n f e r e n c e s b y N I S T
?6 e 4 Z e 4 Z ? ? [ ? * 9 ? ?% Multilingual Subject Gateway (DESIRE) ? ? Developing multilingual subject gateways (SOSIG as example)
SOSIG accept any languages evaluated for quality
Translation should be coherent and checked
Different language version should be equally well maintained
SOSIG Cataloguing rules
TITLE will be displayed in the first language
ALTERNATIVE TITLE in other languages
DESCRIPTION will mention different languages in which resource is available
URI of all language versions
Labeling URI language
Library standards for multilingual provision
NISO Z39.53 Language codes
USMARC Language codes ?Z < ? ? - 1 < ? ? - 1 ? ? k ? + : ? ?: Multilingual provision in popular Internet Search Engines ? ?? Multilingual SE
AltaVista - http://www.altavista.com/ - 28 languages
Documents indexed as is
Automatic translation - very simple and naive
Euroseek - http://www.euroseek.com/ - 30 languages
FAST Advanced Search - http://www.alltheweb.com - 31 languages
Google - http://www.google.com/ - 11 languages
Other sites that have dedicated national sites
interface language
language resources
no special language policy
Excite - 11 countries
Lycos - 23 countries ?~ 6 F ? 0 A + 6 F ? 0 A + ?b ' \ 7 ( ? ?
, ? ?4 TUSTEP TUebingen System of Text Processing Programs ? $ ? ?/ 1. File structure
2. Multilingual capabilities
3. Internal data presentation
4. Database publishing/output data presentation
5. CGI
6. Sample implementation
http://lddv.zdv.uni-tuebingen.de/cgi-bin/opac/zdvlit
Try entries like Smith or Meier or...
http://lddv.zdv.uni-tuebingen.de/cgi-bin/km/npquery
?Z ? 7 & 5 ? 7 & 5 ?? ? 0 ? A I ? ?D Cross-Language Information Retrieval (CLIR) testing at TREC-7/TREC-8 ?
E ? ?? TREC - Text REtrieval Conference - http://trec.nist.gov/
Cross-Language Information Retrieval (CLIR) technologies
Using Intermediary or Interlingual representation
Latent Semantic Indexing
Generalised Vector Space Model, etc.
Computer translation
Machine-readable bilingual dictionaries
MultilingualThesauri
Participants: ETH/Eurospider, IBM, Xerox, Cornell, New Mexico Univ, TNO, others
?Z r 2 > R Q r 2 > R Q ??
" ? 3 ? ?A REIS Project/Initiative Multilinguality framework - First attempt ? ?? M u l t i p l e l a n g u a g e i n d e x i n g
m u l t i p l e l a n g u a g e d o c u m e n t s / i n d e x e s
C r o s s - l a n g u a g e S e a r c h i n g
A u t o m a t i c Q u e r y f o r w a r d i n g b a s e d o n t h e s a u r i o r M L d i c t i o n a r y
U s i n g f u z z y m u l t i l i n g u a l s e a r c h i n g / m a t c h i n g
M u l t i l i n g u a l i n f o r m a t i o n r e t r i e v a l
A u t o m a t i c t r a n s l a t i o n ( i f r e q u e s t e d )
T r a n s l a t i o n R e q u e s t P r o t o c o l
I n t e r n a l D a t a / I n d e x e s p r e s e n t a t i o n
L a n g u a g e a n d C h a r a c t e r E n c o d i n g t a g g i n g
X M L a s i n t e r n a l p r e s e n t a t i o n o f d a t a a n d X M L l a n g u a g e a n d c h a r s e t t a g g i n g
T e x t / C h a r s e t n o r m a l i s a t i o n ( U n i c o d e o r T U S T E P - l i k e )
M e t a d a t a a n d R e s o u r c e D e s c r i p t i o n
D C . L a n g u a g e d e f i n i t i o n a n d X M L / R D F / D C L a n g u a g e t a g g i n g ?? $ l # B $ ? " 7 $ l # B $ ? " 7 ? B J ? ?C Multilinguality Framework for Multilingual Indexing/Search Services ? C ? ? / ? ? ? ? ? ? " ? ' ? ) ? , ? - ? / P ??? b ? r t7 ? J ?
P7
? B b ?k` ?( ` ? ? `
? / ? 0 ? ?0 ?D T i m e s N e w R o m a n ?b ?b D?b ??0h?b h?b ??0 ?D S y m b o l e w R o m a n ?b ?b D?b ??0h?b h?b ??0 ?D M o n o t y p e S o r t s ?b ?b D?b ??0h?b h?b ??0 0 ?D C o u r i e r N e w t s ?b ?b D?b ??0h?b h?b ??0 1 ?
? ? @ @@``?? ?
@ ?n ??? " d d @ ??? ?????? @@ `` ?? 2 ?* ?p ? m ? * & 3 %
$ % ! " #
S ?~ ? ? ? ? ? ? ? ? ??1 ? ? ? ? ? ? ? ? ? ? ? 0 ? ??? ??? n? @ ? ???? ? ?8 ? ? ? ? ?? ?g ? ?4 2 d 2 d t?b ??0l?b > $ ???????? p ? p p ? 0 ? < ?4 B d B d ?b ?b > $ ? ??? ?? ? - ?4 2 0 0 0 . Y u . D e m c h e n k o . T E R E N A ?? M u l t i l i n g u a l I s s u e s i n I n f o r m a t i o n R e t r i e v a l a n d R e s o u r c e D e s c r i p t i o n O ? ? = ?tV ? ? ?S Multilingual Issues in Information Retrieval and Resource Description Overview ?d T + ' ' / + # + ? ?* Yuri Demchenko, TERENA
demchenko@terena.nl ? + + ? 8 A ? ? In this presentation ?
? ?V Multilingual Issues in TERENA Technical Programme
Multilinguality: trends and developments
Technical Issues/Background
Data presentation and resource description format
Standards Overview
Metadata and Cataloging
Recent Development in Subject Gateways and SE
Cross-language Information Retrieval
REIS/TAP Initiatives Multilinguality Framework
?` w E k / w E k / ? V ? 4 < ? ?= TERENA Multilingual Community and TERENA Technical Programme ?
> ? ?: TERENA has 43 members from 34 countries speaking 30 languages
Multilingual issues always were in the scope of TERENA Technical Program
WG-i18n - WG on Internationalisation issues
C3 Project on messaging transliteration tools
MAITS - initiated by WG-i18n
Multilingual E-Mail Agent Testing
Multilingual issues in Subject Gateways, Section 2.13 in SG Handbook
Multilingual Support in Internet/IT Applications. Information page - http://www.terena.nl/projects/multiling/
Liaison with STD bodies
CEN/TC304 Character set technology - http://www.stri.is/TC304/default.html
IETF ?L ? L P ? L P ? ? = E ? ?( Multilinguality: trends and developments ?
) ? ?' Storing, processing, presentation and exchange of information in many languages
Interactive (protocol based/negotiated ) applications and non-interactive (resource description and information presentation)
Multilingual Search and Retrieval
Multilingual Subject Gateways and Search Engines
CLIR testing at TREC
Data Resource Model and Multilinguality
One or Multiple languages
Data format
Metadata (not part of Data but part of Resource)
References, links
Professional Thesauri (Resource Context) - base for multiple languages and language unification
?? P ~ " F ( ? P ~ " F ( ? ?
( ? 6 > ? ? Internet Applications ?
? ?? None-interactive Application: Electronic Mail
Correct Message Composition and Rendering
Interactive applications
WWW: HTTP/HTML
http-equiv="Content-Type" Content="text/html; charset=euc-jp"
Content Negotiation Protocol
Media features, attributes
Direct and hop-by-hop communication
Operational Applications
(Internationalised) DNS
LDAP and X.500 (Language Support ?) ?? . * ? ? < . * ? ? < ?~ ? ,
, ? ? 7 ? ? ?0 I18n and ML issues at IETF and other STD bodies ?
1 ? ?A IETF Architectural Model of Multilingual support in Internet Applications - RFC 2130
Language and Charset/Encoding tagging
Content negotiation framework (IETF/W3C)
Point-to-point vs hop-by-hop
Message based vs Interactive vs Streaming
Internationalised DNS (IDN) - Internationalised Domain Names
vs E-Mail (SMTP, IMAP)
vs Routing (Routing Policy Specification Language (RPSL))
vs Network Management (SNMP textual presentation)
vs Network Security (TLS and IPSec)
Content Encoding normalisation (IETF/Unicode)
LSD-2 - Large Scale Services Deployment
IMAP language extension ?~ U & * G > ? n U & * G > ? n ?? U ^
8 0 2 P ? * ? ?I IETF Architectural Model of Multilingual support in Internet Applications ? ?? User Interface
Presentation
Culture
Locale
Language
On-the-wire
Coded Character Set - Repertoire of ISO-10646
Character Encoding Scheme - UTF-8 (ml-text), US-ASCII (e-mail), ISO8859-1
Transfer Encoding Scheme (Base64, QP)
?Z %
? %
? ? ? ? > F ? ?( Content Negotiation Framework (IETF/W3C) ?
) ? ?? Content Negotiation covers three elements
Expressing the capabilities of the sender and the data resource to be transmitted
Expressing the capabilities of a receiver
A protocol by which capabilities are exchanged
Abstract framework for content negotiation
(Content) (Transmit.data) (Data document)
[Author]----->-----[Sender]----->-----[Receiver]----->-----[User]
Transparent Content Negotiation in HTTP - RFC 2295
Protocol-independent Content Negotiation Framework - RFC 2703
Non-message resource transfer
End-to-end vs hop-by-hop negotiation
Use of directory and resolution services
CC/PP exchange protocol based on HTTP Extension Framework (W3C)
Composite Capability/Preference Profile: A user side framework for content negotiation ?? * ? + ? q l @ W * ? + z q l @ W ? # ? ? + ? ? Charset and Language tagging ? ? MIME types (RFC 2045-2049)
text, img, audio, video
Charset = Character Set + Character Encoding Scheme
Transfer Encoding Scheme
base64
quoted-printable
Other media attributes and features (e.g., resolution, color, language, etc.)
Language
RFC 1766
ISO639-2
?~ e N e N ?, ! ? . ? : B ? ? WWW: HTTP/HTML ? ? ? HTTP header includes information about the type of the transferred information and the character encoding for text-based information:
http-equiv="Content-Type" Content="text/html; charset=euc-jp"
The Content-Language entity header field describes the natural language(s) of the intended audience for the enclosed document:
http-equiv="Content-Type" Content-Language=se
Character encoding information in the META information of the HTML document:
?X ? = ? - N D ?? ? , ? ~ , ? ( 8 ? ? XML: Character Set tagging ? ?? C h a r a c t e r i s a t o m i c u n i t o f t e x t
A l l I S O 1 0 6 4 6 c h a r a c t e r s + T A B , C R , L F
T h e m e c h a n i s m f o r E n c o d i n g c a n v a r y f o r d i f f e r e n t c h a r a c t e r s
A l l X M L p r o c e s s o r s m u s t a c c e p t U T F - 8 a n d U T F - 1 6
C h a r a c t e r E n c o d i n g d e c l a r a t i o n i n X M L d o c u m e n t s o r e n t i t i e s ( s e c t i o n 4 . 3 . 3 )
E n c o d i n g D e c l : : = S e n c o d i n g E q E n c N a m e | E n c N a m e ) < ? x m l e n c o d i n g + U T F - 8 ? > < ? x m l e n c o d i n g + E U C - J P ? >
D e f a u l t C h a r a c t e r S e t E n c o d i n g - U T F - 8 a n d U T F - 1 6
A u t o d e t e c t i o n o f C h a r a c t e r E n c o d i n g ?l ! ' m L ? V ! ' m L ? V ??
G
? ; C ? ? XML: Language tagging ?
? ?? L a n g u a g e i d e n t i f i c a t i o n ( s e c t i o n 2 . 1 2 )
L a b e l l i n g l a n g u a g e o f t h e w h o l e d o c u m e n t , e n t i t y o r i t e m
T a g f o r i d e n t i f i c a t i o n o f l a n g u a g e s
L a n g u a g e I D : : = L a n g c o d e ( - S u b c o d e )
L a n g c o d e : : = I S O 6 3 9 C o d e | I a n a C o d e | U s e r C o d e
E x a m p l e s :
< p x m l : l a n g = " e n " > T h e q u i c k b r o w n f o x j u m p s o v e r t h e l a z y d o g . < / p >
< p x m l : l a n g = " e n - G B " > W h a t c o l o u r i s i t ? < / p >
< p x m l : l a n g = " e n - U S " > W h a t c o l o r i s i t ? < / p >
< s p w h o = " F a u s t " d e s c = ' l e i s e ' x m l : l a n g = " d e " >
< l > H a b e n u n , a c h ! P h i l o s o p h i e , < / l >
< l > J u r i s t e r e i , u n d M e d i z i n < / l >
< l > u n d l e i d e r a u c h T h e o l o g i e < / l >
< l > d u r c h a u s s t u d i e r t m i t h e i ? e m B e m ? h ' n . < / l >
< / s p > ?z ' ] Z
B G ' ] Z
B F ? ?
?
&
$ ? @ H ? ? Unicode Technical Reports ?
? ? The Unicode Standard, Version 3.0 - Just published! - http://www.unicode.org/unicode/uni2book/u2.html
Unicode 2.0 test page http://www.terena.nl/projects/multiling/euroml/tests/test-ucspages1ucs.html
Multilingual European Subsets of ISO/IEC 10646-1 http://www.stri.is/TC304/p10_1998_05_30.pdf
Unicode technical Reports
UTR #15: Unicode Normalization Forms, Version 18.0 I-D by Martin Duerst
UTR #17: Character Encoding Model
UTR #16: UTF-EBCDIC
UTR #10: Unicode Collation Algorithm
UTR #7: Plane 14 Characters for Language Tags ?d g ? ? % / ? ? ?? B Q Z ^ ? ? & 7 ? ?; Language Definition in DC Metadata set - DC.Language Format ? ?? < m e t a n a m e = " D C . L a n g u a g e " c o n t e n t = " e n " >
< m e t a n a m e = " D C . L a n g u a g e " s c h e m e = " r f c 1 7 6 6 " c o n t e n t = " e n " >
< m e t a n a m e = " D C . L a n g u a g e " s c h e m e = " I S O 6 3 9 - 2 c o n t e n t = " e n g " >
< m e t a n a m e = " D C . L a n g u a g e s c h e m e = " r f c 1 7 6 6 c o n t e n t = " e n - U S " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " z h " >
< m e t a n a m e = " D C . L a n g u a g e " c o n t e n t = " j a " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " e s " >
< m e t a n a m e = " D C . L a n g u a g e c o n t e n t = " g e r m a n " >
< m e t a n a m e = " D C . L a n g u a g e l a n g = " f r c o n t e n t = " a l l e m a n d " > ? ? ? ?? ? l ) ) *
? < D ? ?V Language Definition in DC Metadata set - Field content language labelling/attributing ? V ? ?8 A w o r k i n S p a n i s h m a y b e a s s i g n e d t h e f o l l o w i n g m e t a d a t a :
< m e t a n a m e = " D C . L a n g u a g e s c h e m e = " r f c 1 7 6 6 c o n t e n t = " e s " >
< m e t a n a m e = " D C . T i t l e "
l a n g = " e s "
c o n t e n t = " L a M e s a V e r d e y l a S i l l a R o j a " >
< m e t a n a m e = " D C . T i t l e "
l a n g = " e n "
c o n t e n t = " T h e G r e e n T a b l e a n d t h e R e d C h a i r " > ?x t 7 ? 9 @ ? ? DC in Multiple Languages ?
? ?? T h e r e f e r e n c e l a n g u a g e o f I n t l D C c o m m u n i t y i s E n g l i s h , h o w e v e r t h e s e m a n t i c s o d D C e l e m e n t s a r e i n p r i n c i p l e e x p r e s s e d e q u a l l y w e l l i n a n y m o d e r n l a n g u a g e
T h e v e r s i o n s o f D C e l e m e n t s i n v a r i o u s l a n g u a g e s s h o u l d s h a r e a s i n g l e n a m e s p a c e u s i n g t o k e n s t h a t l
???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? o o k l i k e E n g l i s h w o r d s b u t s t a n d f o r u n i v e r s a l e l e m e n t s - h t t p : / / p u r l . o r g / d c / e l e m e n t s / 1 . 1 /
D C i n M u l t i p l e L a n g u a g e s R e g i s t r y p r o j e c t - h t t p : / / p u r l . o r g / d c / g r o u p s / l a n g u a g e s . h t m
U s e s R D F s c h e m a s t o s h a r e m a c h i n e - r e a d a b l e t o k e n s f o r t r a n s l a t i o n o f D C t e r m s i n m u l t i p l e l a n g u a g e s ( 2 6 l a n g u a g e s t o d a t e )
L i n k a g e t o a n d f r o m c e n t r a l D C n a m e s p a c e s e r v e r
R e g i s t r y a s D i c t i o n a r y / T h e s a u r i - u s e I n t e r l i n g u a s t o l i n k d i f f e r e n t t r a n s l a t i o n s
F o r m a l r e c o g n i t i o n a n d s t a n d a r d i z a t i o n p r o c e d u r e ?6 ? ? 1 ? ? 1 ?L O [ ?
P ? ? G ? ?7 Document Description with Unqualified DC and RDF syntax ?
8 ? ?w
Das Erdbeben in Chili
Heinrich von Kleist
XML Encoding (Character set) declaration
UTF-8/UTF-16 as default encoding ?D , + ! + + ! ??
P " { ? . ; ? ?= Recent Developments in Subject Gateways, Indexing, Searching ? ?? N R E N s p r o j e c t s
S u b j e c t g a t e w a y s
C o m m e r c i a l S e a r c h E n g i n e s
M u l t i l i n g u a l T e x t R e t r i e v a l a n d P r o c e s s i n g
T U S T E P s y s t e m - u s i n g f u z z y m u l t i l i n g u a l s e a c h i n g
C r o s s - L a n g u a g e I n f o r m a t i o n R e t r i e v a l ( C L I R ) t e s t i n g a t T R E C - 7 / T R E C - 8 C o n f e r e n c e s b y N I S T
?6 e 4 Z e 4 Z ? ? [ ? * 9 ? ?% Multilingual Subject Gateway (DESIRE) ? ? Developing multilingual subject gateways (SOSIG as example)
SOSIG accept any languages evaluated for quality
Translation should be coherent and checked
Different language version should be equally well maintained
SOSIG Cataloguing rules
TITLE will be displayed in the first language
ALTERNATIVE TITLE in other languages
DESCRIPTION will mention different languages in which resource is available
URI of all language versions
Labeling URI language
Library standards for multilingual provision
NISO Z39.53 Language codes
USMARC Language codes ?Z < ? ? - 1 < ? ? - 1 ? ? k ? + : ? ?: Multilingual provision in popular Internet Search Engines ? ?? Multilingual SE
AltaVista - http://www.altavista.com/ - 28 languages
Documents indexed as is
Automatic translation - very simple and naive
Euroseek - http://www.euroseek.com/ - 30 languages
FAST Advanced Search - http://www.alltheweb.com - 31 languages
Google - http://www.google.com/ - 11 languages
Other sites that have dedicated national sites
interface language
language resources
no special language policy
Excite - 11 countries
Lycos - 23 countries ?~ 6 F ? 0 A + 6 F ? 0 A + ?b ' \ 7 ( ? ?
, ? ?4 TUSTEP TUebingen System of Text Processing Programs ? $ ? ?/ 1. File structure
2. Multilingual capabilities
3. Internal data presentation
4. Database publishing/output data presentation
5. CGI
6. Sample implementation
http://lddv.zdv.uni-tuebingen.de/cgi-bin/opac/zdvlit
Try entries like Smith or Meier or...
http://lddv.zdv.uni-tuebingen.de/cgi-bin/km/npquery
?Z ? 7 & 5 ? 7 & 5 ?? ? 0 ? A I ? ?D Cross-Language Information Retrieval (CLIR) testing at TREC-7/TREC-8 ?
E ? ?? TREC - Text REtrieval Conference - http://trec.nist.gov/
Cross-Language Information Retrieval (CLIR) technologies
Using Intermediary or Interlingual representation
Latent Semantic Indexing
Generalised Vector Space Model, etc.
Computer translation
Machine-readable bilingual dictionaries
MultilingualThesauri
Participants: ETH/Eurospider, IBM, Xerox, Cornell, New Mexico Univ, TNO, others
?Z r 2 > R Q r 2 > R Q ??
" ? 3 ? ?A REIS Project/Initiative Multilinguality framework - First attempt ? ?? M u l t i p l e l a n g u a g e i n d e x i n g
m u l t i p l e l a n g u a g e d o c u m e n t s / i n d e x e s
C r o s s - l a n g u a g e S e a r c h i n g
A u t o m a t i c Q u e r y f o r w a r d i n g b a s e d o n t h e s a u r i o r M L d i c t i o n a r y
U s i n g f u z z y m u l t i l i n g u a l s e a r c h i n g / m a t c h i n g
M u l t i l i n g u a l i n f o r m a t i o n r e t r i e v a l
A u t o m a t i c t r a n s l a t i o n ( i f r e q u e s t e d )
T r a n s l a t i o n R e q u e s t P r o t o c o l
I n t e r n a l D a t a / I n d e x e s p r e s e n t a t i o n
L a n g u a g e a n d C h a r a c t e r E n c o d i n g t a g g i n g
X M L a s i n t e r n a l p r e s e n t a t i o n o f d a t a a n d X M L l a n g u a g e a n d c h a r s e t t a g g i n g
T e x t / C h a r s e t n o r m a l i s a t i o n ( U n i c o d e o r T U S T E P - l i k e )
M e t a d a t a a n d R e s o u r c e D e s c r i p t i o n
D C . L a n g u a g e d e f i n i t i o n a n d X M L / R D F / D C L a n g u a g e t a g g i n g ?? $ l # B $ ? " 7 $ l # B $ ? " 7 ? B J ? ?C Multilinguality Framework for Multilingual Indexing/Search Services ? C ? ? / ?? ? ? ? ? " ? ' ? ) ? , ? - ? / P ??? b ? r A? ? * ?
? ?? B b ??
?????Oh?? +'??0 ?
p x ? ? ? ?
( 4 @
L X ` ?? HTTP и CGI TP = D:\msoffice\Templates\Presentation Designs\International.pot n Yuri Demchenko p 184 Microsoft PowerPoint 7.0 sen@ ???Tv @ ??Ԗѹ?@ `??X7?@ @???? ? G , ???? oZ ? o & ???? ? ? & ???? & # ???? TNPP ?? 0 ?v
&
TNPP ? & ???? & TNPP ??
? ??? - ? ??? " - - ! ? ?? - - ? " - - - - ! ? S?E - - - & ???? t ? ? & ???? - ? - ? ??? - $ t?t?v v ? ??? - ? $ v?v?x x ? ??? - ? $ x?x?{ { ? ??? - ? $ {?{?} } ? ??? - ? $ }?}? ? ??? - ? $ ??? ? ? ??? - ? $ ????? ? ? ??? - ? $ ????? ? ? ??? - ? $ ????? ? ? ??? - ? $ ????? ? ? ??? - ? $ ????? ? ? jj? - ? $ ????? ? ? UU? - ? $ ????? ? ? CC? - ? $ ????? ? ? ,,? - ? $ ????? ? ? ? - ? $ ????? ? - ? - - & ???? & ???? & ???? G k ? & ????
? - ??? ? Times New Roman - ? ??? . . 2
?Q 2000. Yu.Demchenko. TERENA
. & ???? G p y ? & ???? ??? ??? . o 2
?? E Multilingual Issues in Information Retrieval and Resource Description
. & ???? ? n ? ? & ???? ??? ??? . 2
?Z Slide 2
. ??? . 2
?? _ . ??? . 2
?? 1 . - - ZUO 7 - - ??? ??? ? Times New Roman ? - ? . $ 2
? ? Multilingual Issues /
. ??? ? Times New Roman ? - ? .
2
? ? in . ??? ? Times New Roman ? - ? . 2
? ? . ??? ? Times New Roman ? - ? . - 2
? Information Retrieval and
. . % 2
Resource Description
. ??? ? Times New Roman ? - ? . 2
Op Overview . - - 7 ?j - - ??? ? Times New Roman ? - ? . ( 2
?? Yuri Demchenko, TERENA . . $ 2
demchenko@terena.nl
. - - ? ? "System - ? & TNPP & ???? R o o t E n t r y ch_html DEMCH_~2 1 D'Fe desire DESI ???????? ??d?O??? ? ?)? 1 & 1 ??q??? @ ex20C u r r e n t U s e r ???????????? / S u m m a r y I n f o r m a t i o n `? ??n?? ?F8??w? '( ???????? ui*? >???? P?p??? ??h?X?(?h?c
? w? P o w e r P o i n t D o c u m e n t ??G @H? d?}??U?-& G 5??( ????K?=.)??* ??? ?? ? ,?? ??9o??? ? ? ?? ?R?Q
???? ????? ????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ? ????????????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? n ????????? ? ? ? j ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ????? ? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ? _????? ? ? demch d e m c h x ? ? ? ?
( 4 @
L X ` ?? HTTP и CGI TP = D:\msoffice\Templates\Presentation Designs\International.pot n Yuri Demchenko p 184 Microsoft PowerPoint 7.0 sen@ ???Tv @ ??Ԗѹ?@ `??X7?@ @???? ? G , ???? oZ ? . & ???? ? ? & ???? & # ????