{ "info": { "author": "Konrad Weihmann", "author_email": "kweihmann@outlook.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Environment :: Console", "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Natural Language :: English", "Operating System :: POSIX :: Linux", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3 :: Only", "Topic :: Software Development :: Quality Assurance" ], "description": "tlv\n===\n\nToo less variation - A tool to discover code duplication in various\nlanguages\n\nPurpose\n-------\n\nCode duplication are hard to track across many files. There are tools\nexisting like\n`CPD `__ (from PMD)\nwhich are doing a great job, but they require JAVA (or even some more\nobscure language) to run. So I decided to write a small tool which uses\n`Pygments `__ and python\n`difflib `__ to catch\ncode duplications for many languages as possible\n\nRequirements\n------------\n\n- python3\n- `Pygments `__\n\nInstallation\n------------\n\n.. code:: shell\n\n pip install tlv\n\nUsage\n-----\n\n.. code:: shell\n\n usage: tlv [-h] [--minlines MINLINES] [--mintoken MINTOKEN] [-j JOBS]\n [--lexer {...}]\n [--wildcard_token {...}]\n [--verbose] [--nodetails]\n files [files ...]\n\n Find code duplications across various languages\n positional arguments:\n files Files to parse\n\n optional arguments:\n -h, --help show this help message and exit\n --minlines MINLINES Minimum length of block in lines before reporting\n --mintoken MINTOKEN Minimum length of block in token before reporting\n -j JOBS, --jobs JOBS Number of jobs to run in parallel\n --lexer {3d,aap,actionscript,actionscript3,ada,ada2005,ada95,adl,agda,ahk,al,alloy,amienttalk,amienttalk/2,ampl,an,antlr,antlr-actionscript,antlr-as,antlr-c#,antlr-cpp,antlr-csharp,antlr-java,antlr-perl,antlr-python,antlr-r,antlr-ruy,apachecon,apl,applescript,arduino,arexx,as,as3,asemake,asic,asm,aspectj,aspx-cs,asy,asymptote,at,atch,augeas,autohotkey,autoit,awk,c,c#,c++,c++-ojdum,c-ojdump,ca65,cadl,camkes,capdl,capnp,casic,ceylon,chai,chaiscript,chapel,charmci,cheetah,chpl,cirru,cl,clay,clean,clj,cljs,clojure,clojurescript,cmake,cmas,co,code,common-lisp,componentpascal,console,control,cool,coq,cp,cpp,cpp-ojdump,cpsa,cr,crmsh,croc,cry,cryptol,crystal,csh,csharp,csound,csound-csd,csound-document,csound-orc,csound-sco,csound-score,css,css+django,css+er,css+genshi,css+genshitext,css+jinja,css+lasso,css+mako,css+mozpreproc,css+php,css+ruy,css+smarty,cu,cucumer,cuda,cxx-ojdump,cypher,cython,d,d-ojdump,dart,dasm16,decontrol,delphi,desources,dg,di,django,do,docker,dosatch,doscon,dpatch,dtd,duy,dylan,dylan-console,dylan-lid,dylan-repl,e,earl-grey,earlgrey,easytrieve,ec,ecl,eg,ei,elisp,elixir,elm,emacs,emacs-lisp,en,er,erl,erlang,evoque,ex,exs,extempore,ezhil,gap,gas,gawk,genshi,genshitext,gherkin,glsl,gnuplot,go,golo,gooddata-cl,gosu,gro,groovy,gst,haml,handlears,haskell,haxe,haxeml,hexdump,hlsl,hs,hsa,hsail,hspec,html,html+cheetah,html+django,html+er,html+evoque,html+genshi,html+handlears,html+jinja,html+kid,html+lasso,html+mako,html+myghty,html+ng2,html+php,html+ruy,html+smarty,html+spit,html+twig,htmldjango,http,hx,hxml,hxsl,hylang,i,i6t,idl,idl4,idr,idris,iex,igor,igorpro,ik,in,ini,io,ioke,ir,irc,isaelle,itex,j,jade,jags,jasmin,jasminxt,java,javascript,javascript+cheetah,javascript+django,javascript+er,javascript+genshi,javascript+genshitext,javascript+jinja,javascript+lasso,javascript+mako,javascript+myghty,javascript+php,javascript+ruy,javascript+smarty,jcl,jinja,jl,jlcon,jproperties,js,js+cheetah,js+django,js+er,js+genshi,js+genshitext,js+jinja,js+lasso,js+mako,js+myghty,js+php,js+ruy,js+smarty,js+spit,jsg,json,json-ld,json-oject,jsonld,jsp,julia,juttle,kal,kcon,kid,koka,kotlin,lagda,lasso,lassoscript,lcry,lcryptol,lean,less,lhaskell,lhs,lid,lidr,lidris,lighttpd,lighty,limo,liquid,lisp,literate-agda,literate-cryptol,literate-haskell,literate-idris,litzasic,litzmax,live-script,livescript,llvm,logos,logtalk,lua,m2,make,make,mako,maql,mask,mason,mathematica,matla,matlasession,mawk,max,md,minid,mma,modelica,modula2,moin,monkey,monte,moo,moocode,moon,moonscript,mq4,mq5,mql,mql4,mql5,msc,mscgen,mupad,mxml,myghty,mysql,n,nasm,nawk,ncl,nesc,newlisp,newspeak,ng2,nginx,nim,nimrod,nit,nix,nixos,nn,nsh,nsi,nsis,numpy,nusmv,oa,ocaml,octave,odin,oj-c,oj-c++,oj-j,ojc,ojc++,ojdump,ojdump-nasm,ojective-c,ojective-c++,ojective-j,ojectivec,ojectivec++,ojectivej,ojectpascal,ojj,oo,ooc,oogie,opa,openedge,openugs,pacmancon,pan,parasail,pas,pascal,pawn,pcmk,perl,perl6,php,php3,php4,php5,pig,pike,pkgcon,pl,pl6,plpgsql,plus,po,posh,postgres,postgres-console,postgresql,postgresql-console,postscr,postscript,pot,pov,powershell,praat,progress,prolog,properties,protou,ps1,ps1con,psm1,psql,pug,puppet,py,py3,py3t,pycon,pypy,pypylog,pyrex,pyt,python,python3,pyx,qasic,qml,qs,qvt,qvto,r,racket,ragel,ragel-c,ragel-cpp,ragel-d,ragel-em,ragel-java,ragel-ojc,ragel-r,ragel-ruy,rain,raw,rcon,rd,red,red/system,redcode,registry,reol,resource,resourceundle,rexx,rhtml,rkt,rnc,rng-compact,ro,roocon,root,rql,rs,rsl,rts,rust,ruy,s,sage,salt,sarl,sas,sass,satch,sc,scala,scaml,scheme,scila,scm,scss,shell-session,silver,slash,slim,sls,slurm,smali,smalltalk,smarty,sml,snool,sources.list,sourceslist,sp,sparql,spec,spit,splus,sql,sqlite3,squeak,squidcon,st,st-pytex,stan,stata,supercollider,sv,swi,swig,systemverilog,t-sql,tads3,tap,tasm,tcl,tcsh,tcshcon,tea,teraterm,teratermmacro,termcap,termin,terra,thri,todotxt,toml,tra,trac-wiki,treetop,ts,tsql,ttl,turtle,twig,typescript,typoscript,typoscriptcssdata,typoscripthtmldata,ucode,ugs,unicon,uriscript,v.net,vala,vapi,vcl,vclsnippet,vclsnippets,vctreestatus,velocity,vgl,vhdl,vim,vnet,vscript,wdi,whiley,winatch,winugs,x10,xml,xml+cheetah,xml+django,xml+er,xml+evoque,xml+genshi,xml+jinja,xml+kid,xml+lasso,xml+mako,xml+myghty,xml+php,xml+ruy,xml+smarty,xml+spit,xorg,xq,xql,xqm,xquery,xqy,xten,xtend,yaml,yaml+jinja}\n Manually set a lexer to use on all files\n --wildcard_token {Token.Keyword,Token.Keyword.Constant,Token.Keyword.Declaration,Token.Keyword.Namespace,Token.Keyword.Pseudo,Token.Keyword.Reserved,Token.Keyword.Type,Token.Name,Token.Name.Attribute,Token.Name.Builtin,Token.Name.Builtin.Pseudo,Token.Name.Class,Token.Name.Constant,Token.Name.Decorator,Token.Name.Entity,Token.Name.Exception,Token.Name.Function,Token.Name.Function.Magic,Token.Name.Label,Token.Name.Namespace,Token.Name.Other,Token.Name.Tag,Token.Name.Variable,Token.Name.Variable.Class,Token.Name.Variable.Global,Token.Name.Variable.Instance,Token.Name.Variable.Magic,Token.Literal,Token.Literal.Date,Token.Literal.String,Token.Literal.String.Affix,Token.Literal.String.Backtick,Token.Literal.String.Char,Token.Literal.String.Delimiter,Token.Literal.String.Doc,Token.Literal.String.Double,Token.Literal.String.Escape,Token.Literal.String.Heredoc,Token.Literal.String.Interpol,Token.Literal.String.Other,Token.Literal.String.Regex,Token.Literal.String.Single,Token.Literal.String.Symbol,Token.Literal.Number,Token.Literal.Number.Bin,Token.Literal.Number.Float,Token.Literal.Number.Hex,Token.Literal.Number.Integer,Token.Literal.Number.Integer.Long,Token.Literal.Number.Oct,Token.Operator,Token.Operator.Word,Token.Punctuation,Token.Comment,Token.Comment.Hashbang,Token.Comment.Multiline,Token.Comment.Preproc,Token.Comment.Single,Token.Comment.Special,Token.Generic,Token.Generic.Deleted,Token.Generic.Emph,Token.Generic.Error,Token.Generic.Heading,Token.Generic.Inserted,Token.Generic.Output,Token.Generic.Prompt,Token.Generic.Strong,Token.Generic.Subheading,Token.Generic.Traceback,Token.Text.Whitespace}\n Token types that are threated as wildcards (actual\n value doesn't matter)\n --verbose Verbose output\n --nodetails Dump the details of a finding\n\nBy default tool guesses the content type by the filename, if that\ndoesn't work for you please see below\n\nSpecify a lexer\n~~~~~~~~~~~~~~~\n\nYou can use a specific lexer by running the tool with the **--lexer=**\noption. When doing this all input files are processed by the specified\nlexer. So when using this option you should ensure that all passed files\nare of the specified file, else the results might be bogus.\n\nOutput\n------\n\nOutput will be written to stdout. There are 2 possible finding types\n\n- Duplicate - The code is exactly the same\n- TooLessVariation - The code is the same, when removing all token\n types specified by --wildcard\\_token\n\nOutput example\n~~~~~~~~~~~~~~\n\n::\n\n /someplace/busybox/modutils/modutils.c:9:0:[TooLessVariation]:Block till 16:0 is nearly the same as in ../modprobe-small.c from 29:0 till 36:0\n >>> #include \n >>> \n >>> #define init_module(mod, len, opts) syscall(__NR_init_module, mod, len, opts)\n >>> #if defined(__NR_finit_module)\n >>> # define finit_module(fd, uargs, flags) syscall(__NR_finit_module, fd, uargs, flags)\n >>> #endif\n >>> #define delete_module(mod, flags) syscall(__NR_delete_module, mod, flags)\n >>> \n <<<\n >>> #include \n >>> \n >>> #define init_module(mod, len, opts) syscall(__NR_init_module, mod, len, opts)\n >>> #define delete_module(mod, flags) syscall(__NR_delete_module, mod, flags)\n >>> #ifdef __NR_finit_module\n >>> # define finit_module(fd, uargs, flags) syscall(__NR_finit_module, fd, uargs, flags)\n >>> #endif\n >>> \n <<<\n /someplace/busybox/modutils/modutils.c:165:32:[Duplicate]:Block till 169:3 is the same as in ../modprobe-small.c from 324:18 till 327:0\n >>> \n >>> fstat(fd, &st);\n >>> image = NULL;\n >>> /* st.st_size is off_t, we can't just pass it to mmap */\n >>> if\n <<<\n /someplace/busybox/modutils/modprobe-small.c:236:15:[Duplicate]:Block till 240:10 is the same as in ../modutils.c from 265:26 till 269:10\n >>> \n >>> case ENOEXEC:\n >>> return \"invalid module format\";\n >>> case ENOENT:\n >>> return \"\n <<<\n /someplace/busybox/modutils/modutils.c:115:0:[Duplicate]:Block till 130:43 is the same as in ../modprobe-small.c from 177:0 till 194:36\n >>> \n >>> #if ENABLE_FEATURE_CMDLINE_MODULE_OPTIONS\n >>> char* FAST_FUNC parse_cmdline_module_options(char **argv, int quote_spaces)\n >>> {\n >>> char *options;\n >>> int optlen;\n >>> \n >>> options = xzalloc(1);\n >>> optlen = 0;\n >>> while (*++argv) {\n >>> const char *fmt;\n >>> const char *var;\n >>> const char *val;\n >>> \n >>> var = *argv;\n >>> options = xrealloc(options, optlen + 2 + \n <<<\n\nif you don't want to see the code itself, pass the **--nodetails**\noption\n\nFurther reading\n---------------\n\n- `Pygments `__\n\nBugs & Contribution\n-------------------\n\nFeel free to create issues or pull requests\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/priv-kweihmann/tlv", "keywords": "", "license": "", "maintainer": "", "maintainer_email": "", "name": "tlv", "package_url": "https://pypi.org/project/tlv/", "platform": "", "project_url": "https://pypi.org/project/tlv/", "project_urls": { "Homepage": "https://github.com/priv-kweihmann/tlv" }, "release_url": "https://pypi.org/project/tlv/1.0.0/", "requires_dist": null, "requires_python": "", "summary": "-- Too less variation -- Find duplicates in source code for various languages", "version": "1.0.0" }, "last_serial": 5434922, "releases": { "1.0.0": [ { "comment_text": "", "digests": { "md5": "8af074ef23527365a90ea8d210912b2a", "sha256": "5cbfd02a43a1e2b500229b362f33d19cd32edd7a85b9107d1909783e648b15f6" }, "downloads": -1, "filename": "tlv-1.0.0-py3.6.egg", "has_sig": false, "md5_digest": "8af074ef23527365a90ea8d210912b2a", "packagetype": "bdist_egg", "python_version": "3.6", "requires_python": null, "size": 23180, "upload_time": "2019-06-22T15:19:18", "url": "https://files.pythonhosted.org/packages/ac/68/38c5bfb567099a65789c62608feba0cb99848e029811e90360bc4690f606/tlv-1.0.0-py3.6.egg" }, { "comment_text": "", "digests": { "md5": "b9c4c92813cb6a001134753441ea25fc", "sha256": "b0400e7c75bf466b0db19e6a3c42fd1f445e9dd4b0b0595a8cad6a1df33cf33b" }, "downloads": -1, "filename": "tlv-1.0.0.tar.gz", "has_sig": false, "md5_digest": "b9c4c92813cb6a001134753441ea25fc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15387, "upload_time": "2019-06-22T15:20:07", "url": "https://files.pythonhosted.org/packages/4c/08/648a670b05009e67b94dfc4e49e9c1d0cebfaa8835dfb278baa155af43fe/tlv-1.0.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "8af074ef23527365a90ea8d210912b2a", "sha256": "5cbfd02a43a1e2b500229b362f33d19cd32edd7a85b9107d1909783e648b15f6" }, "downloads": -1, "filename": "tlv-1.0.0-py3.6.egg", "has_sig": false, "md5_digest": "8af074ef23527365a90ea8d210912b2a", "packagetype": "bdist_egg", "python_version": "3.6", "requires_python": null, "size": 23180, "upload_time": "2019-06-22T15:19:18", "url": "https://files.pythonhosted.org/packages/ac/68/38c5bfb567099a65789c62608feba0cb99848e029811e90360bc4690f606/tlv-1.0.0-py3.6.egg" }, { "comment_text": "", "digests": { "md5": "b9c4c92813cb6a001134753441ea25fc", "sha256": "b0400e7c75bf466b0db19e6a3c42fd1f445e9dd4b0b0595a8cad6a1df33cf33b" }, "downloads": -1, "filename": "tlv-1.0.0.tar.gz", "has_sig": false, "md5_digest": "b9c4c92813cb6a001134753441ea25fc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15387, "upload_time": "2019-06-22T15:20:07", "url": "https://files.pythonhosted.org/packages/4c/08/648a670b05009e67b94dfc4e49e9c1d0cebfaa8835dfb278baa155af43fe/tlv-1.0.0.tar.gz" } ] }